What is a robots.txt File?
The robots.txt file acts as the primary gatekeeper for your website. Placed directly in the root directory of your host (e.g., https://yourdomain.com/robots.txt), it tells Search Engine Crawlers which URLs they are allowed to scrape and index, and which directories (like admin dashboards or database backup folders) they are strictly forbidden from viewing.
How to use the Generator
- Select the specific search engine you wish to control. Leaving it as `*` applies the rule universally to all bots.
- Choose either `Allow` or `Disallow`. By default, unless specified, bots consider all public pages allowed.
- Specify the trailing path structure. For example, entering `/cart/` will prevent Google from attempting to index thousands of individual user checkout pages.
- Attach the URL of your XML sitemap at the bottom so crawlers know exactly where to find your most important content.
Warning: Malicious human attackers will often read your robots.txt file to find your hidden admin directories. Never rely on robots.txt as a security measure for preventing human hacking. Always utilize proper server-side authentication.