The robots.txt file is a text file placed in your site's root directory (e.g., /public_html/) that tells search engine bots which pages to crawl and which to ignore .
- How to use: You can create this file via the Hostaday File Manager.
- Example: Use
User-agent: *followed byDisallow: /admin/to prevent bots from indexing your private folders. - Caution: Misconfiguring this file can hide your entire website from Google, so use it carefully .
