Free AI Crawler Access Test
Your robots.txt can welcome ChatGPT while your firewall turns it away. This free test visits your homepage as 15 real crawlers, including GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Googlebot, and shows which ones actually get in.
Free · No sign-up · Nothing stored
robots.txt is the sign on the door. Your firewall is the guard
robots.txt asks crawlers to stay out of certain pages. Your firewall, CDN or bot protection decides who actually gets in, and it can block AI crawlers by name even when robots.txt welcomes them. The crawler then gets an error or a challenge page instead of your content, and AI tools can't read or cite you. This happened on our own site, and no robots.txt checker could see it. That's why this test visits as each crawler instead of reading a file.
How the AI crawler access test works
- A normal visit first. We load your homepage as an ordinary browser, to know what a real visitor gets.
- Then 15 crawler visits. We load the same page with each crawler's real user agent.
- Compare. Each visit is judged against the normal one: same content, an error, a challenge or a near-empty page.
- Cross-check robots.txt. We show what your robots.txt allows next to what your server actually did.
The 15 crawlers, in three groups
| Group | Crawlers | Why it matters |
|---|---|---|
| Search engines | Googlebot, Bingbot | Feed Google, Bing and the AI answers built on them (AI Overviews, ChatGPT search, Copilot) |
| AI search and assistants | OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Applebot | Decide whether ChatGPT, Claude, Perplexity and Siri can show and cite your pages |
| AI training | GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot | Collect content to train models; blocking them is a business choice |
What each result means
| Result | What we saw | What to do |
|---|---|---|
| Allowed | HTTP 200 and about the same content a browser gets | Nothing |
| Blocked | 401, 403, 429 or 503 | Allow the crawler in your firewall or CDN |
| Challenged | A bot-check page instead of your content | Exempt the crawler from the challenge |
| Different | Far less content than a normal visitor | Make sure content is in the HTML, not added only by JavaScript |
| robots.txt mismatch | Let in by the server but disallowed by robots.txt, or the reverse | Decide which you mean and align both |
Real result: our site today
When we first built this test, our own CDN's bot protection was turning AI crawlers away while our robots.txt welcomed them. After switching that setting off for the crawlers we want, all 15 visits to cmoowls.com come back allowed: both search engines, all seven AI search and assistant crawlers, and the six training crawlers. If you'd rather keep training crawlers out, block them deliberately in robots.txt and keep the AI search crawlers allowed.
Search, assistants or training: choosing who gets in
The three groups do different jobs, so they deserve different decisions. Search engine crawlers feed Google and Bing, and the AI answers built on them. AI search and assistant crawlers fetch pages when someone asks ChatGPT, Claude or Perplexity a question, so blocking them removes you from those answers. Training crawlers collect content to build future models; blocking them is a reasonable choice that doesn't affect AI search. The Robots.txt Generator builds a file that keeps search and assistants in and training crawlers out, if that's what you want.
What this test can and can't tell you
- We test the name, not the address. Firewalls that verify crawlers by IP address may treat the real crawler differently, often correctly blocking a fake Googlebot. If only Googlebot or Bingbot look blocked, confirm in Search Console and Bing Webmaster Tools.
- Homepage only. Rules that apply to specific folders won't show here.
- A moment in time. Bot settings can change with any dashboard update; re-test after changes.
AI crawler access on one map
For what each crawler does and who runs it, read What Is an AI Crawler?, and for the full picture of being found by AI assistants, how to check your AI visibility. The AI Visibility Checker reads your robots.txt rules for 18 crawlers.
Check every AI visibility signal, not just access
CMO Owls audits your SEO and AI visibility across every page, and Generative Engine Owl suggests the fixes.
Frequently asked questions
Is the AI Crawler Access Test free?
Yes. It's free, with no sign-up and nothing stored.
How is this different from the AI Visibility Checker?
The AI Visibility Checker reads your robots.txt, which says who is welcome. This test actually visits your site as each crawler, which shows who your firewall, CDN or bot protection lets in. You need both: robots.txt can say yes while the firewall says no.
Why does Googlebot show as blocked when Google indexes my site fine?
Many firewalls verify Googlebot and Bingbot by their real IP addresses and block anyone using their names from elsewhere. Our test uses their names from our server, so a block here can mean your firewall is doing its job. Confirm with URL Inspection in Google Search Console and in Bing Webmaster Tools.
What does "challenged" mean?
The crawler got a bot-protection page, such as a "Just a moment" check, instead of your content. Crawlers can't solve these, so to them your site is empty.
How do I unblock AI bots in my CDN?
Look for bot settings in your CDN or firewall dashboard, such as bot-fight modes, "block AI bots" switches and custom rules that match crawler names, and allow the AI search crawlers you want: OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot. In Cloudflare these live under Security, then Bots, and in WAF custom rules.
Should I block AI training crawlers?
That's a business choice. Blocking GPTBot, ClaudeBot or CCBot stops your content being used for model training, but it doesn't remove you from AI search answers, which use separate crawlers.
What is a user agent?
The name a browser or crawler sends with every request, such as GPTBot or Googlebot. Firewalls and robots.txt rules use it to decide how to treat the visitor. This test sends each crawler's real user agent.
Why does a crawler get a much smaller page than visitors?
Usually because the content is added by JavaScript, which most AI crawlers don't run, or because the server sends crawlers a stripped-down version. Either way, that crawler can't read what your visitors see.
How often should I run this test?
After any change to your CDN, firewall, hosting or security plugins, and every few months otherwise. Bot settings change with dashboard updates, and new AI crawlers appear.
Agents that do this for you
Related reading