Key Takeaways
- An AI crawler is a bot run by an AI company that fetches web pages, either to train models, to build an AI search index, or to read a page live because a user asked about it.
- Those three jobs use different crawlers. Blocking GPTBot (training) doesn't remove you from ChatGPT search, which uses OAI-SearchBot.
- Most AI crawlers read the raw HTML your server sends and don't run JavaScript. Content that only appears after scripts load can be invisible to them.
- robots.txt is a request; your firewall or CDN decides who actually gets in. Check both, because they often disagree.
What Is a Web Crawler?
A web crawler (also called a spider or bot) is a program that visits web pages automatically, reads them, and follows their links to find more pages. Search engines have run crawlers since the 1990s: Googlebot and Bingbot fetch billions of pages so Google and Bing can index them and show them in results.
Every crawler announces itself with a user agent, a short text string sent with each request, such as Googlebot/2.1 or GPTBot/1.2. Before visiting a site, a well-behaved crawler reads the site's robots.txt file to see which paths it may visit.
What Is an AI Crawler?
An AI crawler is a web crawler operated by an AI company such as OpenAI, Anthropic, Perplexity, Google, Apple or Meta. It works the same way as a search crawler, but what happens to your pages afterwards is different. A search crawler builds an index of links. An AI crawler's pages become training data for a model, sources in an AI search index, or the text an assistant reads aloud to answer a question in real time.
So, can AI read websites? Yes: when you ask ChatGPT, Claude or Perplexity about a page or a product, they usually search their index and then fetch the relevant pages live. What they can't read is anything their crawler can't reach: pages behind a login or paywall, pages your firewall blocks, and content that only exists after JavaScript runs.
The Three Kinds of AI Crawlers
This is the distinction most robots.txt files get wrong. AI companies split their crawlers by job, and each job has a different effect on your visibility:
- Training crawlers collect public pages to train future models. Blocking them keeps your content out of training data. It doesn't, on its own, remove you from AI answers.
- AI search crawlers build the index that AI assistants search when they answer questions and cite sources. Blocking them removes your pages from those answers.
- User-triggered fetchers visit a page at the moment a person asks the assistant to read it or follows up on a result. Providers describe these as acting on behalf of the user. OpenAI's crawler documentation says plainly that for ChatGPT-User, "robots.txt rules may not apply."
AI Crawlers List: Who Visits Your Site
Here are the AI crawlers and AI-related user agents you'll see most often in server logs, grouped by job:
| Crawler (user agent) | Company | What it does |
|---|---|---|
| GPTBot | OpenAI | Collects content to train OpenAI models (training) |
| OAI-SearchBot | OpenAI | Indexes pages so they can appear in ChatGPT search results (AI search) |
| ChatGPT-User | OpenAI | Fetches a page when a ChatGPT user's request needs it (user-triggered) |
| ClaudeBot | Anthropic | Collects content to train Claude models (training) |
| Claude-SearchBot | Anthropic | Indexes pages for Claude's search results (AI search) |
| Claude-User | Anthropic | Fetches pages when a Claude user asks (user-triggered) |
| PerplexityBot | Perplexity | Indexes pages for Perplexity answers (AI search) |
| Perplexity-User | Perplexity | Fetches pages to answer a user's question (user-triggered) |
| Google-Extended | Not a separate crawler: a robots.txt token that controls whether Google may use your content for Gemini. It doesn't affect Google Search or AI Overviews (Google's crawler list) | |
| Applebot / Applebot-Extended | Apple | Applebot powers Siri and Spotlight; Applebot-Extended controls use for Apple's AI training |
| CCBot | Common Crawl | Builds a free public web archive that many AI models are trained on |
| Meta-ExternalAgent | Meta | Collects content for Meta's AI models |
| Bytespider | ByteDance | Collects content for ByteDance's AI models |
| Amazonbot | Amazon | Used for Alexa answers and Amazon's AI services |
Two crawlers deserve an extra word. Googlebot isn't on this list, but it matters for AI anyway: Google's AI Overviews and AI Mode draw on the regular Google index, so blocking Googlebot removes you from those too. And Bing's index feeds Microsoft Copilot and is widely used by other AI search products, which is why Bingbot and Bing Webmaster Tools belong in any AI visibility plan.
What is Common Crawl?
Common Crawl is a nonprofit that has crawled the web since 2008 and publishes the results as a free archive, petabytes of pages that anyone can download. Its crawler is CCBot. Because the archive is free and huge, it's one of the most widely used sources of AI training data: OpenAI's GPT-3 paper, for example, reports that a filtered version of Common Crawl made up 60% of its training mix. Blocking CCBot keeps future crawls of your site out of that archive.
How AI Crawlers Read a Page
When an AI crawler visits, the sequence is usually short:
- It reads robots.txt at the root of the site and looks for a group that names it, falling back to the
User-agent: *group. - It requests the page and receives the HTML your server sends, plus the HTTP status code (200, 301, 403 and so on).
- It extracts the text, usually without running JavaScript. An analysis of crawler traffic published by Vercel in December 2024 found that none of the major AI crawlers it studied render JavaScript: they fetch script files as text but don't run them.
- It stores or uses the content: in a training dataset, in a search index, or directly in an answer.
Step three is the one that surprises people. Googlebot renders JavaScript, so a site built entirely in client-side JavaScript can still rank in Google. To most AI crawlers, the same site can look like an empty shell with a loading spinner. If your product descriptions, prices or articles are injected by scripts, render them on the server or pre-render them. The same applies to schema markup: JSON-LD added by a tag manager after load may never be seen by AI crawlers.
Crawling vs. scraping vs. indexing
| Term | What it means |
|---|---|
| Crawling | Visiting pages automatically and following links to discover more |
| Scraping | Extracting specific data from pages, such as prices or reviews, often for reuse |
| Indexing | Storing and organizing what was crawled so it can be searched or retrieved later |
AI tools can scrape too. Developers use crawling frameworks and headless browsers to feed pages into AI applications, and ChatGPT can read pages you point it to. Whether a particular scrape is allowed depends on the site's terms, the data involved and the law where you are, not on the technology.
How to Control AI Crawlers
You have four levers, and they work at different levels:
| Lever | What it controls | Strength |
|---|---|---|
| robots.txt | Which crawlers may visit which paths | A request. Respected by the major AI companies |
| Firewall, CDN or bot protection | Who can connect at all | Enforced. Overrides robots.txt |
| noindex and nosnippet | Whether pages appear in search results, and how much text can be shown or quoted | Enforced by search engines on crawlable pages |
| llms.txt | Which pages you'd like AI tools to read first | A guide, not a gate. It can't block anything |
Allow AI search, opt out of training
Most businesses want to be cited in AI answers. If you also want to keep your content out of model training, name the training crawlers in their own group and leave the search crawlers on the default rules:
User-agent: *
Disallow: /admin/
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /
Sitemap: https://www.example.com/sitemap.xmlOAI-SearchBot, Claude-SearchBot and PerplexityBot aren't named, so they follow the * group and can read your public pages. The free Robots.txt Generator builds and verifies a file like this from your current one. Our guide to robots.txt covers the rules in depth.
Cloudflare and AI crawlers
This is where most surprises come from. Cloudflare offers settings that block AI crawlers by user agent, and since July 2025 it blocks known AI crawlers by default on newly added domains. Other CDNs and security plugins have similar switches. When one is on, the crawler gets a 403 error or a challenge page, no matter what your robots.txt says.
It happened to us. Our robots.txt welcomed GPTBot and ClaudeBot, but a Cloudflare setting answered both with 403 Forbidden. No robots.txt checker could see it, because the problem wasn't in robots.txt. That's why we built a test that visits your site as each crawler.
What is crawler access?
Crawler access simply means whether a given crawler can reach and read your pages: robots.txt allows it, the server returns your real content, and nothing in between (firewall, CDN, login wall, JavaScript) gets in the way. On hosted platforms the controls live in the platform. Shopify, for example, generates your robots.txt automatically and lets you edit it through a `robots.txt.liquid` theme template.
Access Is the First Layer of AI Visibility
In our 57-factor framework for AI visibility, access is layer one, and every later layer (content, entity, off-site, commerce, measurement) depends on it. If a crawler can't get in, nothing else you publish can be cited. These are the access factors, and how to check each one:
| Factor | What to check |
|---|---|
| robots.txt | Search and AI search crawlers aren't blocked. Check it |
| Firewall and bot protection | AI crawlers get your page, not a 403 or a challenge. Test it |
| CDN | Edge rules and caching don't serve crawlers a different page |
| Rendering | Key text is in the raw HTML, not only after JavaScript runs |
| Ungated content | Important pages aren't behind logins, pop-up walls or paywalls |
| XML sitemaps | Every public page is listed and the sitemap is in robots.txt |
| IndexNow | New and changed pages are pushed to Bing and others instantly |
| Bing Webmaster Tools | The site is verified and the sitemap submitted |
| Indexing | Important pages are indexed in Google Search Console |
| Snippets | No accidental nosnippet or tiny max-snippet limits on pages you want quoted |
| Canonicals | Each page points to the right preferred URL |
| llms.txt | A clean summary of your key pages for AI tools. Learn more |
| Server logs | You can see which AI crawlers actually visit, and what status they get |
Broken links belong here too: crawlers discover pages by following links, and a dead link is a dead end. Our guide to broken links covers how to find and fix them.
How to Check Which AI Crawlers Can Access Your Site
- Read your robots.txt rules per crawler with the free AI Visibility Checker, which tests 18 crawlers and also checks your llms.txt and homepage.
- Test the server's response per crawler with the free AI Crawler Access Test. It visits your homepage as 15 crawlers and compares each response with a normal browser visit, so firewall blocks and challenge pages show up.
- Check your server or CDN logs for the user agents in the list above, and the status codes they received.
- Verify suspicious bots. User agents can be faked. Google documents reverse-DNS verification for Googlebot, and OpenAI publishes the IP ranges its crawlers use, so you can confirm a request really came from them.
One caveat on testing tools: a test that sends a crawler's user agent from its own server can only check rules based on the name. Firewalls that verify real crawler IP addresses may treat the genuine crawler differently, and often correctly block a fake Googlebot. If only Googlebot or Bingbot look blocked, confirm with URL Inspection in Google Search Console and in Bing Webmaster Tools.
Should You Block AI Crawlers?
Block training crawlers if you don't want your content used to train models; it's a business decision, and a reasonable one for publishers and creators. Think hard before blocking AI search crawlers or user-triggered fetchers: they're how ChatGPT, Claude and Perplexity find, read and cite you. A growing share of people now ask an assistant before they search, and a site those assistants can't read can't be recommended.
Also keep in mind that robots.txt depends on good behavior. The major AI companies say they respect it, but in August 2025 Cloudflare reported that it had seen Perplexity reaching sites with undeclared crawlers after its declared bots were blocked, which Perplexity disputed. If you truly need content kept away from bots, a firewall rule or a login is the only real control.
FAQ
What is an AI crawler in simple terms?
It's a bot from an AI company that visits websites to read their pages, either to train AI models, to build an AI search index, or to answer a user's question in real time.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects content to train OpenAI's models. OAI-SearchBot indexes pages for ChatGPT search. Blocking GPTBot doesn't remove you from ChatGPT search results; blocking OAI-SearchBot does.
Can ChatGPT read websites?
Yes. ChatGPT searches its index and fetches pages live when a question needs them. It can't read pages behind a login, pages your firewall blocks, or content that only appears after JavaScript runs.
Do AI crawlers follow robots.txt?
The major AI companies say their crawlers follow robots.txt. User-triggered fetchers act on behalf of a person, and some providers say robots.txt may not apply to them the same way. Use a firewall rule for anything that must be blocked.
Do AI crawlers run JavaScript?
Mostly not. Analyses of crawler traffic show the major AI crawlers read the HTML a server sends without executing scripts. Googlebot is the notable exception. Put important content in the server-rendered HTML.
How do I block AI crawlers?
Add their user agents to robots.txt with Disallow: /, and if you need enforcement, block them in your firewall or CDN. Block training crawlers only if you want to stay visible in AI search.
What is Common Crawl used for?
Common Crawl publishes a free archive of the web that researchers and AI companies use, including as a major source of training data for large language models. Its crawler is CCBot.
Why does my site block AI crawlers when robots.txt allows them?
Usually a firewall, CDN or security plugin setting, such as Cloudflare's AI bot controls. It refuses the request before robots.txt is ever considered. Test your site's real responses with the AI Crawler Access Test.
Keep AI Crawlers Seeing the Right Thing
Crawler access breaks during redesigns, CDN changes and plugin updates, usually without anyone noticing. Owl-gorithm Hacker audits your robots rules and compares raw and rendered HTML across your site, so JavaScript-only content gets caught, and Generative Engine Owl checks the signals that come next: schema, brand consistency and AI discoverability. To check both layers yourself, read how to check your AI visibility.
Sources
- Overview of OpenAI crawlers: GPTBot, OAI-SearchBot, ChatGPT-User and their IP ranges.
- Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic Help Center.
- Perplexity crawlers, Perplexity documentation.
- Google's common crawlers, including Google-Extended, Google Search Central.
- Verifying Googlebot and other Google crawlers, Google Search Central.
- Common Crawl and Language Models are Few-Shot Learners (GPT-3), Brown et al., 2020.
- Content Independence Day: no AI crawl without compensation, Cloudflare, July 2025.
- Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives, Cloudflare, August 2025.
- The rise of the AI crawler, Vercel, 2024.
- Customize robots.txt, Shopify developer documentation.