Robots.txt Generator & Validator
Create crawler rules for the whole site or specific bots, add a sitemap, and check an existing robots.txt file for syntax problems. Nothing is uploaded.
Robots.txt rules
Start with a preset or build per-bot groups. The output updates as you edit.
Presets replace the groups below. Choosing Custom keeps your current groups.
Use an absolute http:// or https:// URL. The Sitemap directive is added after the user-agent groups.
Add a user-agent group or sitemap URL to generate robots.txt.
Errors
Warnings
Generated robots.txt
Upload the file to your site root, usually at /robots.txt.
No rules generated yet.
What the generated file does
- User-agent groups apply rules to one crawler or to
*for all crawlers. - Allow and Disallow paths support
*wildcards and$end-of-URL anchors. - Presets can allow everything, block a staging site, or block a list of common AI crawlers.
- Sitemap tells crawlers where to find your sitemap, but it does not grant access.
Private by design: the generator and validator run locally in your browser. No robots.txt content is sent to a server.
Validate robots.txt
Paste a file to check structure, directives, paths, and sitemap URLs.
Paste a robots.txt file or validate the generated output.
Validation results
No validation results yet.
What robots.txt can and cannot do
Robots.txt is a crawler instruction file, not an access-control system. It can ask well-behaved crawlers not to fetch certain paths, but it cannot prevent a person or a rogue bot from requesting a public URL. Do not use robots.txt to hide passwords, private customer data, admin panels, or anything that must stay confidential. Protect those resources with authentication and server-side permissions.
User-agent groups and rule order
A group starts with one or more User-agent lines and is followed by Allow, Disallow, or Crawl-delay rules. The wildcard * matches every crawler without a more specific group. Crawlers choose the most specific matching rule; when an Allow and a Disallow rule are equally specific, the more permissive rule is commonly used. Keep groups simple and test the result in the validator.
Common robots.txt mistakes
Noindexis not a robots.txt directive. Use a meta robots tag or anX-Robots-TagHTTP header.- A sitemap URL must be absolute and must point to an actual sitemap file.
- Blocking JavaScript, CSS, or image paths can prevent search engines from rendering and understanding pages correctly.
- Blocking every crawler while also submitting a sitemap is usually contradictory.
- Crawl-delay is not supported by Google, even though some other crawlers honor it.
Deploying robots.txt
Upload the file to the root of your domain so it is available at https://example.com/robots.txt. Do not place it in a subdirectory unless that directory is the site root. After deployment, open the URL directly, confirm that it returns a 200 response with the expected text, and validate the live file again if you made changes.