Robots.txt Generator

Create a robots.txt file with rules for all crawlers or specific bots, sitemap links and ready-made presets — including one that blocks AI crawlers such as GPTBot, ClaudeBot and Google-Extended. Test any path against your rules before you publish.

This tool runs entirely in your browser.

Loading tool…

How to use the Robots.txt Generator

1Start from a preset or edit the rule groups: user agents, paths to disallow and exceptions to allow.
2Add your sitemap URL.
3Test a few paths, then download robots.txt and upload it to the root of your site.

What is the Robots.txt Generator?

robots.txt sits at the root of a domain and tells crawlers which paths they may fetch. Rules are grouped by user agent; a crawler uses the most specific group that names it and ignores the rest, so a Googlebot group replaces the * rules for Google entirely. Within a group the longest matching rule wins, and Allow wins a tie — so Allow: /private/open overrides Disallow: /private/. * matches any characters and $ anchors the end, as in Disallow: /*.pdf$.

The tester follows these rules (Google’s interpretation, standardised in RFC 9309). Remember that robots.txt is a request, not access control: well-behaved crawlers honour it, but it doesn’t hide content, and a blocked URL can still be indexed from links — use noindex for that.

Frequently asked questions

How do I block ChatGPT and other AI crawlers?

Use the “Block AI crawlers” preset. It disallows GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot and others while leaving normal search engines allowed.

Does Crawl-delay work?

Bing and Yandex honour it; Google ignores it and adjusts its crawl rate automatically.

Is the Robots.txt Generator free and private?

Yes. The Robots.txt Generator is free, needs no account, and runs entirely in your browser. What you type is never sent to a server or stored.

All developer tools →