AI Crawler Access Auditor

Audit whether AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended +more) can crawl your site, and generate the correct robots.txt + llms.txt. Free, no API key.

AI Crawler Access Checker with three states, not two

Fetches your robots.txt, resolves it the way Google's specification says a parser must, and reports each of twelve AI crawlers as explicitly allowed, explicitly blocked, or inheriting the wildcard group. That third state is the one every other tool collapses into green. It matters, because per Google's spec a named group and the * group are never combined.

Three states, not allowed/blockedTwelve named bots, current tokensTraining vs search vs user-fetchVendor doc quoted per bot

Built and reviewed by Sayed Hasan, founder of SEOs Hut · Updated 28 September 2026 · Free, no signup

Is GPTBot blocked on my site?

Check for a group with the literal token GPTBot. If one exists, that group alone applies — Google's robots.txt spec states user-agent specific groups and global groups are not combined, so your Allow: / under * is irrelevant to it. With no named group, GPTBot inherits the wildcard rules. Blocking it affects model training only, not search.

The reference

Twelve AI crawlers: token, purpose, compliance, search effect

Tokens are exact and case-insensitive in matching. Purposes are the vendors' own words, not our categorisation. Where a vendor publishes nothing, the cell says so rather than guessing — which is why one row is grey.

Crawler	robots.txt token	Documented purpose	Obeys robots.txt	Affects search ranking or inclusion
OpenAI — GPTBot	`GPTBot`	Training OpenAI's generative AI foundation models. UA contains `GPTBot/1.4; +https://openai.com/gptbot`.	Yes	NoTraining only. OpenAI documents no search effect, and Google does not use it as a signal. Blocking it does not touch your rankings.
OpenAI — OAI-SearchBot	`OAI-SearchBot`	Search only.	Yes	Yes — in ChatGPT"Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." The one documented case where a block costs you visibility.
OpenAI — ChatGPT-User	`ChatGPT-User`	User-initiated fetches, when a person asks ChatGPT to look at something.	Partly"Because these actions are initiated by a user, robots.txt rules may not apply."	No"is not used to determine whether content may appear in Search." A fetch by this agent is evidence of a fetch and nothing else.
OpenAI — OAI-AdsBot	`OAI-AdsBot`	Validating ad landing pages. UA `OAI-AdsBot/1.0`. Its data "is not used to train generative AI foundation models."	Yes	NoAdvertising verification. No documented organic effect on any surface.
Anthropic — ClaudeBot	`ClaudeBot`	Training.	YesAnthropic also supports the non-standard `Crawl-delay` extension, which Google ignores.	NoTraining. No documented search effect.
Anthropic — Claude-User	`Claude-User`	User-initiated fetches.	Not statedAnthropic lists it as a user-initiated fetcher without publishing a robots.txt exemption. We will not infer one.	NoUser-driven retrieval, not indexing.
Anthropic — Claude-SearchBot	`Claude-SearchBot`	Indexing to improve search result quality.	Yes	Within Anthropic's surfacesDocumented as indexing for search quality. Nothing to do with Google's ranking.
Perplexity — PerplexityBot	`PerplexityBot`	"Designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models."	Yes	Yes — in PerplexityBlocking it removes you from the surface it feeds. Explicitly not a training crawler.
Perplexity — Perplexity-User	`Perplexity-User`	User-requested fetches.	No"Since a user requested the fetch, this fetcher generally ignores robots.txt rules."	NoYour robots.txt answer for this row is, by Perplexity's own account, largely academic.
Google — Google-Extended	`Google-Extended`	Control token for Gemini Apps and Vertex AI training and grounding. It "doesn't have a separate HTTP request user agent string… the robots.txt user-agent token is used in a control capacity."	Yes — control onlyNo fetches arrive under this name. You will never see it in a log.	No"Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."
Apple — Applebot-Extended	`Applebot-Extended`	Control token governing how data crawled by Applebot may be used. It "does not crawl webpages".	Yes — control only	NoPages disallowing it "can still be included in search results", and "Site rules for Applebot-Extended are not considered in ranking for Search."
Meta — Meta-ExternalAgent	`Meta-ExternalAgent`	"Training foundation AI models or improving products by indexing content directly." UA `meta-externalagent/1.1`.	Yesrobots.txt cached for up to 24 hours. Note that the separate `Meta-ExternalFetcher` "may bypass robots.txt rules".	Not documentedMeta publishes no statement linking it to search ranking. The newer `Meta-WebIndexer` exists so Meta AI can "cite and link to your content".
ByteDance — Bytespider	`Bytespider`	UndocumentedUA is `Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com)`. No reachable vendor documentation states its purpose.	UndocumentedWe will not claim it ignores robots.txt. We have no vendor source either way, and neither does anyone stating otherwise.	UndocumentedNo published statement exists.

Scroll the table sideways to see every column.

You may have seen Claude-Web and anthropic-ai on other bot lists. They are deprecated and still widely republished as current. Rules naming them are harmless and do nothing.

The rule everything else follows from

Wildcard and named groups are never merged

“Only one group is valid for a particular crawler… the group with the most specific user agent that matches… Other groups are ignored. User agent specific groups and global groups (*) are not combined.”
<b>Google, robots.txt specification, updated 31 August 2026</b> — Google Search Central

Parser behaviour

Six spec details that change what your file actually does

Read the file the way a parser does, not the way a document reads. Each of these has produced a production incident somebody had to explain in a meeting.

Behaviour	What the spec says	Why it changes your answer
Group selection	One group only — the most specific matching user-agent token. Every other group is ignored for that crawler.	Reading your `*` rules tells you nothing about a bot that has its own group. This is the entire reason the third state exists in our output.
No inheritance	Named groups and the global `*` group are not combined.	`Allow: /` under `` does not rescue a bot whose own group disallows everything. And your `Disallow: /wp-admin/` under `` does not apply to any bot with a named group — so an over-eager named group can open a directory you thought was closed.
File size	Google parses the first 500 KiB and ignores the remainder.	A rule beneath that line does not exist. Plugin-generated files with hundreds of bot groups reach it more often than you would expect.
Caching	Cached for up to 24 hours.	Your fix is not live when you deploy it. OpenAI documents roughly 24 hours before its systems adjust; Meta states up to 24 hours as well. Re-testing immediately measures the cache.
crawl-delay	Unsupported by Google. Anthropic supports it as a non-standard extension.	The same line is honoured by one vendor and ignored by another. Never use it as your only rate control.
Error responses	A 4xx response, except 429, is treated as no restrictions at all.	A robots.txt returning 404 or 403 is a fully open robots.txt, not a locked one. Confirm the file actually serves 200 before you conclude that anything is blocked.

Scroll the table sideways to see every column.

Conflicting rules resolve to the least restrictive match, and the longest matching path wins. A specific Allow: therefore beats a broad Disallow: — which is why skimming the top of a file and stopping at Disallow: / produces the wrong answer so reliably.

Two agents make your robots.txt advisory at best

Perplexity states of Perplexity-User: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." OpenAI states of ChatGPT-User: "Because these actions are initiated by a user, robots.txt rules may not apply." Meta documents that Meta-ExternalFetcher "may bypass robots.txt rules". These are vendor statements, not accusations. The practical consequence: if your requirement is that a specific page is never retrieved by a chat product on a user's behalf, robots.txt is the wrong layer. Authentication is the right layer. Anything else is a request.

Decide once, then write the file

Do you want to be cited, or not trained on?

These are separate decisions with separate tokens and separate costs. Most self-inflicted blocks come from answering one and editing for the other.

Do you want your pages cited and linked in AI answers?

Yes — citations and referral clicks are the point

Allow the search fetchersExplicitly allow OAI-SearchBot, PerplexityBot, Claude-SearchBot and Meta-WebIndexer. Leave Googlebot untouched. Then make sure there is something worth lifting once they arrive — allowed in is step one, being useful once inside is step two, which is the AEO Readiness Checker.

No — you sell access to the content itself

Block search fetchers knowinglyDisallow OAI-SearchBot and accept the documented consequence: you will not be shown in ChatGPT search answers. Nothing about your Google ranking changes.

Do you object to your text being used for model training?

Yes, but you still want search and AI citations

Block training, allow searchDisallow GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and Meta-ExternalAgent while allowing the search fetchers. Both Google and Apple state their control tokens carry no ranking consequence, so this combination costs you nothing documented.

You have blocked Googlebot to keep out AI Overviews

Self-inflicted, and expensiveUndo it. Google states AI is built into Search and integral to how Search functions, so there is no Googlebot rule that removes AI features while keeping Search. Use Google-Extended for the training and grounding objection, and nosnippet to limit what is shown.

You do not object, and want maximum reach

Nothing to blockWrite explicit Allow: / groups anyway, so the next person to read the file knows it was a decision. Then give the fetchers a map of what is worth reading with the llms.txt Generator — modest value, and honest about it.

robots.txt — allow search, disallow training

# Search and citation fetchers: explicitly allowed.
# Each named group must restate any path rules you still want.
User-agent: OAI-SearchBot
Disallow: /wp-admin/
Allow: /

User-agent: Claude-SearchBot
Disallow: /wp-admin/
Allow: /

User-agent: PerplexityBot
Disallow: /wp-admin/
Allow: /

User-agent: Meta-WebIndexer
Disallow: /wp-admin/
Allow: /

# Model-training and grounding: explicitly disallowed.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

# Googlebot is deliberately absent from the blocks above:
# AI is built into Search, so blocking it removes you from Search.
User-agent: *
Disallow: /wp-admin/
Disallow: /cart/
Allow: /

Note the repeated Disallow: /wp-admin/ inside every named group. That is not redundancy — named groups do not inherit from *, so omitting it would permit those four fetchers into a directory the wildcard group protects. Two caveats this file cannot express: ChatGPT-User and Perplexity-User are documented as possibly or generally ignoring robots.txt, so they are absent by design rather than oversight, and expect up to 24 hours before any of this takes effect.

From real files

Four robots.txt errors we find most often

None of these is exotic. All four survive in production because the file looks correct when you read it as English.

Reading the wildcard group and stopping

What happens: User-agent: * / Allow: / at the top, so the conclusion is that everything is allowed. Meanwhile a named group 200 lines down disallows GPTBot entirely, and per the spec that group is the only one GPTBot reads.

Do this instead: Resolve per bot, most-specific-token-first, the way a parser does. That is the whole job of this tool, and it is why our output has three states rather than a green tick.

Writing a named group and forgetting its path rules

What happens: A shiny new User-agent: OAI-SearchBot / Allow: / group. The site's Disallow: /checkout/ lives under *, is not inherited, and that fetcher now has the run of the checkout flow.

Do this instead: Copy every path rule you care about into each named group. Verbose files are fine; silently widened permissions are not.

Blocking training crawlers to protect rankings

What happens: Google-Extended and GPTBot get disallowed as a defensive SEO measure. Google states Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal", and Apple says its own rules are not considered in ranking. The move is neutral for SEO, which means the reasoning was wrong even when the outcome is harmless.

Do this instead: Block training because you have a view about training. It is a licensing decision, not a ranking one. Keep the two arguments separate when the stakeholder asks.

Calling the job done at the permissions layer

What happens: The file is perfect and the logs show nothing fetching anyway — or show fetchers arriving and leaving with an almost empty page, because the content needs JavaScript to exist.

Do this instead: Permissions are the theory; logs are what happened. Run your access log through the AI Crawler Log Analyzer to see which agents actually arrived, confirm Googlebot's separate story with the Google Index Checker, and check that the page has content without JavaScript — a JS-dependent page can be fully allowed and still effectively empty, which the Core Web Vitals Checker will show you.

The two-line version for your stakeholder

Blocking training crawlers has no documented effect on Google or Apple search rankings — both vendors say so in their own documentation, quoted above with links. The one block with a documented visibility cost is OAI-SearchBot, which removes you from ChatGPT search answers. Everything else in this argument is somebody's inference. Bring the matrix to the meeting, and then go and read the logs — the whole permissions question is theoretical until you know which agents are actually turning up. Run a visibility check afterwards to see whether any of it changed a real answer.

Questions

AI Crawler Access Checker FAQ

Is GPTBot blocked on my site?

Look for a group whose user-agent token is literally GPTBot. If one exists, that group alone governs it — Google's spec states named and wildcard groups are not combined. With no named group, GPTBot follows your * rules. Our output distinguishes those two cases, because an explicit allow and a silent inheritance are different facts.

Should I block AI crawlers?

That depends on whether you want your content used for training, which is your call and not ours. Mechanically: blocking training crawlers has no documented search cost, and blocking OAI-SearchBot removes you from ChatGPT search answers. Those are the two facts. Everything else being argued about this is inference.

Does blocking GPTBot hurt SEO?

No. GPTBot is documented as a training crawler for OpenAI's foundation models and has no relationship to Google's index. Separately, Google states Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal", and Apple says Applebot-Extended rules are not considered in ranking for Search.

What is Google-Extended in robots.txt?

A control token, not a crawler. Google documents that it "doesn't have a separate HTTP request user agent string" — the token is used in a control capacity only, governing whether data may be used for Gemini Apps and Vertex AI training and grounding. You will never see it in an access log, and blocking it carries no ranking consequence.

How do I allow ChatGPT to crawl my site?

Add an explicit group for OAI-SearchBot with Allow: /, and restate any path rules you want kept, since named groups do not inherit from *. Allow GPTBot too if you have no objection to training. Then wait — OpenAI documents around 24 hours before its systems reflect a robots.txt change.

Does User-agent: * with Allow: / allow every AI crawler?

No. Per Google's robots.txt specification, only the most specific matching group applies and "user agent specific groups and global groups (*) are not combined". Any bot with its own named group is reading that group alone. Your wildcard Allow is invisible to it.

Is Claude-Web still a current Anthropic bot?

No. Claude-Web and anthropic-ai are deprecated, and several popular bot lists still print them as current. The current tokens are ClaudeBot for training, Claude-User for user-initiated fetches, and Claude-SearchBot for search indexing. Anthropic also publishes verification IP ranges at claude.com/crawling/bots.json.

Which AI bots ignore robots.txt?

By their own documentation: Perplexity-User "generally ignores robots.txt rules", ChatGPT-User's rules "may not apply" because the fetch is user-initiated, and Meta-ExternalFetcher "may bypass robots.txt rules". If a page must never be fetched, use authentication. Bytespider is frequently accused of this; no vendor documentation exists either way, so we report it as undocumented.

Primary sources used on this page

Google robots.txt specification: one valid group per crawler, named and wildcard groups not combined, 500 KiB limit, 24-hour caching, crawl-delay unsupported, 4xx handling — developers.google.com
OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot — purposes, independence of settings, and the ~24-hour lag — developers.openai.com
Anthropic: ClaudeBot, Claude-User and Claude-SearchBot, Crawl-delay support and bot verification IPs — support.claude.com
Perplexity: PerplexityBot is search-only, and Perplexity-User generally ignores robots.txt — docs.perplexity.ai
Google: Google-Extended is a control token with no impact on Search inclusion or ranking — developers.google.com
Apple: Applebot-Extended does not crawl, and its rules are not considered in ranking for Search — support.apple.com
Meta: Meta-ExternalAgent, Meta-WebIndexer, Meta-ExternalFetcher and the 24-hour robots.txt cache — developers.facebook.com
Google: AI is built into Search, and the snippet-level controls that limit what AI features show — developers.google.com

Keep going

From permissions to what actually happened

robots.txt states an intention. These four tell you whether the fetchers arrived, whether they found anything, and whether it made any difference to an answer.

AI Crawler Log Analyzer llms.txt Generator Google Index Checker AEO Readiness Checker AI Visibility Checker Core Web Vitals Checker All 26 free tools →

Read the guide: How to Find the Sitemap of a Website, Robots.txt Best Practices: 10 Rules for SEO and Does llms.txt Work? What Google and AI Bots Really Do With It. Want it handled for you? See our AI SEO (GEO) services.