Free Robots.txt Generator

What is robots.txt?

robots.txt is a plain text file at the root of your website (yoursite.com/robots.txt) that tells crawlers which parts of the site they may visit. Search engines and AI crawlers read it before they crawl. It's a request, not a lock: well-behaved crawlers follow it, but it doesn't hide or protect anything.

In the age of AI search, robots.txt does two jobs at once. It keeps private areas like your admin, cart and checkout out of search, and it decides which AI crawlers can read your public pages. One wrong line can quietly remove you from ChatGPT, Claude or Perplexity answers.

How this robots.txt generator works

1. It reads your current file

Already have a file? This works as a free robots.txt checker and tester too. We load your existing robots.txt, check which search engines and AI crawlers it allows, and keep your rules for any other crawlers, so nothing you rely on is lost. We also flag lines that search engines silently ignore, such as a misspelled field name or a rule placed before any User-agent line.

2. It looks for private areas and subdomains

We check your site for common private sections, including admin areas, dashboards, account pages, carts, checkouts, internal search and API paths, and select the ones that actually exist so they stay out of search. Sign-up and registration pages are never selected for you, because many sites want those found. We also check common subdomains like app, dashboard and staging, which need their own robots.txt, and we recognize wildcard DNS so we don't invent subdomains that aren't real.

3. It verifies the result before you copy it

We test the generated file with the same matching rules search engines use: every crawler you allowed can reach your site, every crawler you blocked can't, your private sections are closed to all crawlers, and none of the public pages in your sitemap are blocked by mistake. We also flag blocked CSS or JavaScript files, and add your sitemap if the file doesn't list one.

Robots.txt syntax, in plain words

Three rules decide what happens when lines conflict. A crawler with its own group follows only that group and ignores the * group. Within a group, the longest matching rule wins. If an Allow and a Disallow rule are the same length, Allow wins. These are the rules in RFC 9309 and Google's documentation, and they're the rules this generator verifies against.

Example: a robots.txt that stays visible in AI search

This example keeps private areas out of search, lets search engines and AI search crawlers in, and opts out of AI training. Blocking training is a business choice; the Recommended preset does the same.

# Search engines and AI search crawlers: allowed everywhere public
User-agent: *
Disallow: /admin/
Disallow: /account
Disallow: /cart
Disallow: /checkout

# AI training crawlers: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Notice the training crawlers have their own group, so they follow only Disallow: /. OAI-SearchBot, ChatGPT-User, PerplexityBot and the others aren't named, so they follow the * group and can read every public page.

AI crawlers explained

Common robots.txt mistakes

• Blocking everything by accident: User-agent: * with Disallow: / often ships from a staging site and blocks every crawler.
• Blocking CSS or JavaScript: search engines need them to render your pages properly.
• Typos: a misspelled field like Dissallow is silently ignored.
• Using robots.txt to hide pages: it's public and only a request. Use a login, or a noindex tag.
• Disallowing a page you want removed from Google: if crawlers can't visit it, they can't see its noindex tag.
• Forgetting subdomains: app.yoursite.com and staging.yoursite.com each need their own robots.txt.

Test it after you upload

Use this page as a robots.txt tester: once the new file is live, analyze your site again and confirm every setting matches. Then run the free AI Visibility Checker to confirm all 18 crawlers see the access you intended, and check Google's view in the robots.txt report in Google Search Console. Remember that a firewall can still block AI crawlers that robots.txt allows. Our guide How to Check Your AI Visibility shows the one-line test.