Robots.txt Generator
Create a clean robots.txt file for your website using User-agent, Allow, Disallow and Sitemap directives. Build multiple crawler rule groups, review the output and download a ready-to-use robots.txt file.
Robots.txt Generator
Build a valid robots.txt with crawler rule groups and sitemaps, then copy or download it.
Your robots.txt rules are generated locally in your browser and are not uploaded to or stored on our servers. No account is required.
How to use Robots.txt Generator
Choose a preset or start a group
Apply a safe preset, or build your own rule group from scratch.
Add Allow and Disallow paths
Set the paths each user-agent may or may not crawl.
Add sitemap URLs
Optionally list one or more absolute sitemap URLs.
Generate and export
Generate the file, review it, then copy or download robots.txt.
Key Features
- Multiple user-agent groups: Create separate rule groups for different crawlers.
- Allow and Disallow directives: Add as many path rules as you need per group.
- Sitemap and crawl-delay support: List multiple sitemaps and set an optional crawl-delay.
- Safe presets: Allow-all, block-all, basic site, WordPress, and block-a-crawler starting points.
- Validation and warnings: Blocking errors and advisory warnings are shown separately.
- Browser-based, copy and download: Everything runs locally; download a ready robots.txt. No signup.
Frequently Asked Questions
What is robots.txt?
robots.txt is a text file at the root of a site that tells crawlers which paths they may or may not crawl. It guides well-behaved crawlers but does not enforce access.
Where should robots.txt be uploaded?
Place it at the root of your domain, for example https://example.com/robots.txt. Crawlers look for it there.
Does robots.txt prevent indexing?
No. It controls crawling, not indexing. A blocked URL can still appear in search results, usually without a description, if it is linked elsewhere.
Can robots.txt protect private pages?
No. robots.txt is not a security mechanism. Anyone can read it, and it does not stop access. Use authentication for private data.
What is User-agent: *?
The asterisk is a wildcard that applies the rule group to all crawlers that do not have their own more specific group.
What is the difference between Allow and Disallow?
Disallow asks a crawler not to crawl matching paths, while Allow permits paths, often to create an exception within a disallowed area.
Can I add more than one sitemap?
Yes. You can list multiple Sitemap lines, each pointing to an absolute URL. They typically appear at the end of the file.
Does Google support crawl-delay?
Crawl-delay is supported by some crawlers, but not all search engines interpret it the same way, and some ignore it. Treat it as advisory.
What happens if robots.txt contains an error?
Crawlers may skip lines they cannot parse, and behavior varies. Test important rules before relying on them, since syntax support differs between crawlers.
Is the generated file uploaded or stored?
No. The robots.txt is generated entirely in your browser and is never uploaded to or stored on our servers.