How to use the robots.txt generator
- Choose your rule: allow all crawlers, or disallow specific paths.
- If disallowing, list one path per line, such as /admin or /tmp.
- Optionally enter your sitemap URL and a crawl delay in seconds.
- Click Generate robots.txt, then copy the result into a file named robots.txt at the root of your site.
How robots.txt works
A robots.txt file sits at the root of your domain and tells well-behaved crawlers which parts of your site they may access. It is a set of instructions, not a security control — a crawler can ignore it, and the file does not hide pages from people who already know the URL. Search engines such as Google generally respect the rules when deciding what to crawl and index.
Key directives explained
| Directive | Purpose |
|---|---|
| User-agent | Which crawler the following rules apply to. The * wildcard matches all of them. |
| Disallow | A path the crawler should not visit. |
| Allow | An exception that re-opens a path inside a disallowed folder. |
| Crawl-delay | Seconds a crawler should wait between requests. |
| Sitemap | The absolute URL of your XML sitemap. |
What to block and what to leave open
Most sites block private or low-value areas such as admin panels, search result pages, staging copies and login screens. You generally want public content, images and CSS available so pages render and rank properly. Blocking everything is rarely a good idea, and blocking stylesheets can stop Google from rendering your page at all.
Robots.txt best practices
Keep your file small and readable, and remember that directives are case-sensitive and order matters for some crawlers. Always link your sitemap from the file so crawlers find every page you want indexed, and place the file in the root directory only — a robots.txt in a subfolder is ignored. Remember that blocking a page with Disallow does not remove it from an index; it only stops future crawls. Finally, test your rules after publishing by requesting the file in a browser and checking coverage reports in your search console to confirm the paths you intended to block are actually excluded.