AI Crawler Log Analyzer
Paste your server log to see which AI bots actually crawled you, how often, and which pages. Free, no API key.
AI Crawler Log Analyzer for the GPTBot line you can't tell is real
You grepped your access log, found a few thousand hits claiming to be GPTBot, and now you need to know three things: is it real, is it the fetcher that actually matters for citation, and has your WAF been silently blocking it for a month. Paste your log lines and we'll separate the bots, bucket the status codes, and flag the CDN header trap that fails every genuine hit if you check the wrong field. If you haven't set your access rules yet, do that first — this tool reads what already happened.
Built and reviewed by Sayed Hasan, founder of SEOs Hut · Updated 28 September 2026 · Free, no signup
Does seeing GPTBot in my logs mean my content was used to train a model?
Not necessarily, and check the token first. GPTBot is OpenAI's training crawler; OAI-SearchBot and ChatGPT-User are separate tokens for answer-time fetching and user-initiated fetches. A GPTBot hit tells you about training crawl access, not about whether you were ever cited — those run on different bots with different published IP ranges.
Reverse DNS does not verify an AI crawler the way it verifies Googlebot
Only Google and Apple publish a reverse-DNS method. OpenAI, Anthropic and Perplexity publish CIDR files and nothing else — there is no hostname to resolve for GPTBot, ChatGPT-User, ClaudeBot or PerplexityBot. So 'reverse-DNS it' is simply wrong advice for the bot most people search this page to verify. And behind a CDN, checking the wrong field fails every genuine hit anyway: origin logs show the edge's IP in remote_addr, not the crawler's, so a CIDR check against that field returns 'unverified' for traffic that's completely real.
Every AI bot token, and how to actually verify it
Fetched from each operator's own published file. Match the real client IP — not remote_addr behind a CDN — against the CIDR file, or the rDNS mask where one exists.
| Token | Purpose | Obeys robots.txt | rDNS mask? | CIDR file | Last updated |
|---|---|---|---|---|---|
| OAI-SearchBot | Answer-time citation fetch | Yes, ~24h to pick up robots.txt changes | No — CIDR only | openai.com/searchbot.json (39 prefixes) | 2026-01-02 |
| GPTBot | Training crawl | Yes | No — CIDR only | openai.com/gptbot.json (18 prefixes) | 2026-09-22 |
| ChatGPT-User | User-initiated fetch (someone pasted your URL) | May not apply — user-triggered, not automatic | No — CIDR only | openai.com/chatgpt-user.json (230 prefixes) | 2026-09-25 |
| OAI-AdsBot | Ads-related fetch | Not detailed by OpenAI | No — CIDR only | openai.com/adsbot.json | Not stated |
| ClaudeBot / Claude-User / Claude-SearchBot | Training, answer-time and search fetch — one combined list for all three | Supports the non-standard Crawl-delay extension; Anthropic warns IP blocking may not persist | No — CIDR only | claude.com/crawling/bots.json (26 prefixes, mixed cloud ranges) | 2026-08-18 |
| PerplexityBot | Training / indexing crawl | Not detailed by Perplexity | No — CIDR only | perplexity.com/perplexitybot.json (8 prefixes) | 2025-02-07 — 19 months stale |
| Perplexity-User | User-initiated fetch | Generally ignores robots.txt — match User-Agent AND IP together, per Perplexity's own guidance | No — CIDR only | perplexity.com/perplexity-user.json (4 prefixes) | 2025-10-17 |
| Googlebot / Google-Agent / Google-GeminiNotebook | Crawl, agent action, and notebook fetch | Yes | Forward-confirmed rDNS in googlebot.com / google.com / googleusercontent.com | 5 files under developers.google.com/static/crawling/ipranges | Rolling, no fixed date |
| Applebot | Search crawl | Follows Googlebot's rules if none are named for Applebot specifically; no crawl-delay support | rDNS in *.applebot.apple.com, or CIDR match | search.developer.apple.com/applebot.json | Not stated |
| Bingbot | Search crawl | Not detailed here | Not stated here | bing.com/toolbox/bingbot.json | 2024-01-03 — over two years stale |
Scroll the table sideways to see every column.
Applebot-Extended does not crawl webpages at all and will never appear in a server log — if you're trying to control training use, it's a robots.txt directive, not a bot you can log. Google is also experimenting with Web Bot Auth, cryptographic request signing under the identity https://agent.bot.goog, a different verification model from either CIDR matching or rDNS.
Three numbers that explain why a hardcoded IP list goes stale
Why this hit fails a naive CIDR check, and shouldn't
Reading the real client IP instead of the edge's
# Cloudflare's published edge ranges — add the full current list
set_real_ip_from 173.245.48.0/20;
set_real_ip_from 103.21.244.0/22;
set_real_ip_from 103.22.200.0/22;
# ...repeat for every range Cloudflare currently publishes
real_ip_header CF-Connecting-IP;
real_ip_recursive on;This only fixes what nginx writes to $remote_addr in your own log — it doesn't touch the verification step itself. You still take that corrected IP and check it against the crawler's own CIDR file (openai.com/gptbot.json, claude.com/crawling/bots.json, and so on), never against Cloudflare's ranges.
Five ways a verification script gets it wrong
Reverse-DNS'ing a GPTBot or ClaudeBot hit
What happens: OpenAI, Anthropic and Perplexity publish CIDR files and nothing else — there's no documented hostname to resolve for any of their bots. A script built around rDNS returns nonsense for exactly the operators most people are trying to verify.
Do this instead: Match the IP against that operator's own published CIDR file. Save rDNS for Googlebot and Applebot, the two operators that actually document it.
Checking remote_addr against a CIDR file behind a CDN
What happens: Behind Cloudflare, an ALB, or any reverse proxy, $remote_addr at your origin is the proxy's IP, not the crawler's. Every genuine hit fails the check, which looks exactly like a spoofed bot until you know to look elsewhere.
Do this instead: Read CF-Connecting-IP, True-Client-IP, or the correct hop of X-Forwarded-For — whichever your specific proxy sets — before running the CIDR match.
Treating every published CIDR file as equally fresh
What happens: Bing's bingbot.json is dated 2024-01-03. Perplexity's own PerplexityBot file is 19 months old. A hardcoded copy of any of these, cached indefinitely, silently drifts out of date — while ChatGPT-User's file, 12x larger than GPTBot's, gets updated within days.
Do this instead: Fetch each file at verification time rather than hardcoding a copy, and note the creationTime field the operator publishes.
Counting a frequent ChatGPT-User hit as a citation
What happens: OpenAI documents ChatGPT-User as a user-initiated fetch — someone pasted your URL into the chat box. It isn't automatic crawling and robots.txt rules may not even apply to it. A spike in these hits is evidence people are sharing your link, not evidence you were cited in an answer.
Do this instead: Report OAI-SearchBot and ChatGPT-User as two separate rows. They prove two different things.
Assuming a robots.txt error means the bot is blocked
What happens: RFC 9309 says the opposite of what most people assume: a 4xx response fetching robots.txt means the crawler MAY access everything, and only a 5xx means it MUST assume complete disallow.
Do this instead: Check the actual status code your server returns for /robots.txt, not just whether a browser can load the file.
Six terms this topic mixes up constantly
- Token
- The exact string inside the User-Agent header — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, and so on. There is no such thing as "ChatGPTBot"; that string doesn't exist in any operator's published list.
- CIDR file
- A JSON list of IP prefixes an operator publishes for a given bot, refreshed on its own schedule per file. The only verification method for OpenAI, Anthropic and Perplexity's bots.
- rDNS mask
- A reverse-DNS hostname pattern an operator documents for forward-confirmed verification. Published only by Google and Apple; nobody else offers it.
- Edge IP vs client IP
- Behind a CDN or load balancer, remote_addr in your origin log is the proxy's IP. The real client IP lives in a header — CF-Connecting-IP, True-Client-IP, or a specific hop of X-Forwarded-For.
- Crawl-delay
- A non-standard robots.txt directive, not part of RFC 9309, that Anthropic's bots specifically support. Most operators ignore it.
- Web Bot Auth
- Google's experimental cryptographic request-signing protocol, under the identity https://agent.bot.goog — a verification model separate from both CIDR lists and rDNS.
What to actually run this afternoon
Grep your log for the token, not a guess at the UA string. Then find the real client IP — read the CDN header, don't trust remote_addr — and match it against that specific operator's CIDR file, fetched fresh, not a cached copy from six months ago. If the volume looks wrong once you've done that, check your actual access rules, then see whether a change in fetch volume lines up with a change in your visibility trend. And if the bots are burning fetches on thin URLs, point them at a shorter, better list; if they're burning fetches on redirect chains or hitting dead pages, that's usually a site-structure problem, not a bot-management one.
AI Crawler Log Analyzer FAQ
Why Do AI Crawlers Keep Hitting robots.txt Instead of My Content?
Why do I see AI bot traffic but no crawling of fresh content?
How can I access server logs for SEO purposes?
What is the difference between CDN logs and origin server logs for AI bot analysis?
Why don't I see any ChatGPT-User hits at all?
Is there such a thing as ChatGPTBot?
Will Applebot-Extended show up in my logs?
Primary sources used on this page
- OpenAI: the four published bots and their CIDR files — platform.openai.com
- Anthropic: ClaudeBot, Claude-User, Claude-SearchBot, Crawl-delay support, and the IP-blocking warning — support.claude.com
- Perplexity: PerplexityBot and Perplexity-User CIDR files, and the UA+IP matching guidance — docs.perplexity.ai
- Google: the forward-confirmed reverse-DNS verification procedure — developers.google.com
- Google: Google-Agent, Google-GeminiNotebook, and the Web Bot Auth experiment — developers.google.com
- Apple: Applebot verification methods and the Applebot-Extended training opt-out — support.apple.com
- RFC 9309: robots.txt parsing limits and the 4xx/5xx disallow rules — www.rfc-editor.org
Turn the log data into a fix
A verified bot hit is the start of the job, not the end of it. These six are where the fetch data actually leads.
Read the guide: What Is Crawl Budget? How It Works and When It Matters and What Is Crawling in SEO? How Search Engines Crawl a Site. Want it handled for you? See our AI SEO (GEO) services.