Free CEO Strategy Call

AI Crawler Access Checker

By /

AI Crawler Access Auditor

Audit whether AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended +more) can crawl your site, and generate the correct robots.txt + llms.txt. Free, no API key.

Free · no API key · checks if AI bots can reach your site + generates the fix

Powered by SEOs Hut
Home / Free Tools / AI Crawler Access Checker
Free tool · No signup

AI Crawler Access Checker

Enter your URL and see whether eight named AI crawlers are allowed or blocked in your robots.txt. You get a 0-100 score, a count of how many AI search engines can reach you, and a ready-to-paste robots.txt block plus a starter llms.txt. Free, no signup.

Checks 8 AI crawlersTraining vs search splitCopy-paste robots.txtStarter llms.txt included

How do I check if AI crawlers can access my website?

Open your robots.txt file and look for rules naming AI crawlers such as GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and PerplexityBot. A Disallow rule under any of those names blocks that bot. This free checker reads the file for you, gives a verdict for each of eight crawlers, and generates a corrected robots.txt block you can paste in.

The key split

Training crawlers and search crawlers are not the same thing

This is the single most important idea on this page, and it is easy to get wrong. Some AI bots collect text to train models. Others fetch pages so they can cite you in an answer. Blocking the first group is a fair business decision. Blocking the second group takes you out of AI answers completely, and that can happen by accident with one blanket rule.

1

Search crawlers: let these in

OAI-SearchBot, Claude-SearchBot, PerplexityBot and ChatGPT-User fetch pages so an AI can link to you. OpenAI says OAI-SearchBot does not collect training data. If you block these, you are not protecting your content, you are removing yourself from the answers your customers see.

2

Training crawlers: your call

GPTBot, ClaudeBot, Google-Extended and Applebot-Extended govern whether your text is used to train models. Publishers with original research often block them. For a local plumber or dentist, there is usually little to lose and some upside in leaving them open. The checker reports them separately so you decide.

3

Google-Extended is not Googlebot

Google-Extended is a robots.txt token that controls whether your content trains Gemini. Google states it does not affect your inclusion in Google Search, and AI Overviews are built from Googlebot's crawl. So blocking Google-Extended does not keep you out of AI Overviews, and it does not hurt your rankings.

4

The blanket-block trap

A rule reading User-agent: * followed by Disallow: / blocks everything, including the search crawlers that would have cited you. Plugins and security tools sometimes add AI-blocking rules during setup without making the training and search difference clear. The checker shows the verdict per bot so nothing hides.

The eight crawlers we check

Each line is the exact user-agent name to use in your robots.txt file.

OpenAI search crawler - allow
User-agent: OAI-SearchBot
OpenAI live page fetch - allow
User-agent: ChatGPT-User
Anthropic search crawler - allow
User-agent: Claude-SearchBot
Perplexity crawler - allow
User-agent: PerplexityBot
OpenAI training crawler - your call
User-agent: GPTBot
Anthropic training crawler - your call
User-agent: ClaudeBot
Google AI training token - your call
User-agent: Google-Extended
Apple AI training token - your call
User-agent: Applebot-Extended
The line that follows each one
Allow: /
What you get

A score, a verdict list, and files you can paste

The output is meant to be usable in five minutes, not studied. You get a 0-100 access score, a yes or no for each of the eight crawlers, a plain count of how many AI search engines can currently reach you, and two generated files. Copy the robots.txt block over your current AI rules and you are done.

1

The generated robots.txt block

We build a block that names each AI crawler explicitly, allows the search bots, and leaves the training bots as a choice you make. Paste it into your existing robots.txt rather than replacing the whole file, so your other rules and sitemap line survive.

2

The starter llms.txt

We also generate a simple llms.txt listing your key pages. Be realistic about it: Google has said it does not use llms.txt for Search or AI Overviews, and no major AI company has formally committed to reading it in production. It is cheap insurance, not a ranking lever.

3

Where to put the file

Robots.txt has to sit at the root of your domain, so it loads at yoursite.com/robots.txt. On WordPress many SEO plugins have a robots.txt editor built in. On other platforms you upload it to the public folder. If it loads anywhere else, crawlers will not find it.

Questions

AI Crawler Access Checker FAQ

Should I block AI crawlers from my website?
For most local businesses, no. Blocking search crawlers like OAI-SearchBot, Claude-SearchBot and PerplexityBot removes you from AI answers entirely, so a customer asking ChatGPT for a recommendation will never see you. Blocking training crawlers like GPTBot is a separate decision with no direct effect on whether AI engines can cite your pages, so you can block those and still appear.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot is OpenAI's training crawler, introduced in August 2023, which collects publicly available content for future models. OAI-SearchBot is OpenAI's search crawler: it finds and indexes pages so ChatGPT can link to them in answers, and OpenAI says it does not collect training data. They are controlled separately in robots.txt, so you can allow one and block the other.
Will blocking Google-Extended hurt my Google rankings?
No. Google-Extended is a robots.txt token that controls whether your content is used to train Gemini and related generative products. Google states it does not affect your inclusion in Google Search and is not a ranking signal there. AI Overviews are built from Googlebot's crawl, so blocking Google-Extended does not remove you from AI Overviews either.
Why does Applebot-Extended never show in my logs?
Because it is not a crawler. Applebot-Extended is a robots.txt token that tells Apple how it may use content already collected by the standard Applebot. It never makes a request, so it will never appear in an access log. Google-Extended works the same way. That is why a robots.txt check and a log check answer different questions.
I do not have a robots.txt file at all. Is that bad?
Not usually. With no robots.txt, crawlers assume they may fetch anything public, so AI search crawlers can already reach you. The risk is the opposite one: with no file, you also have no record of your intent, and a plugin or host can later add a blanket block without you noticing. An explicit file makes your choices visible.
Do AI crawlers actually obey robots.txt?
The major named ones say they do. OpenAI, Anthropic and Google all publish their user-agent names and state that their crawlers respect robots.txt rules. Anthropic confirms all three of its bots honor it, including the one that fetches a page when a user asks. Robots.txt is a request, not a wall, so it will not stop badly behaved scrapers.
How often should I re-run this check?
After any change to your hosting, your SEO plugin, your security plugin or your CDN, and then roughly once a quarter. Robots.txt rules get rewritten by tools more often than people expect, and a block added during a migration can sit there unnoticed for months. The check takes a few seconds, so there is no reason to skip it.
Keep going

Other free tools worth a minute

AEO Readiness CheckerAI Crawler Log Analyzerllms.txt GeneratorAll free tools →

Not sure what your robots.txt should say?

Run the check, then send us the result. On a free strategy call we will tell you which crawlers to allow for your business and why, without selling you a service you do not need.

Book a free CEO strategy call →
Scroll to Top