AI Crawler Checker: Can ChatGPT, Claude & Perplexity See You?

Reference

The AI crawlers this tool checks

Each line restates the company's own documentation, checked on 7 November 2026. The crawler names are the tokens you write in robots.txt.

Search and AI answers

Crawlers that decide whether search engines and AI assistants can find and cite your pages.

Crawler	What it does	Source
`OAI-SearchBot`OpenAI	Finds pages for ChatGPT search results.	OpenAI docs
`Claude-SearchBot`Anthropic	Crawls to improve the quality of Claude's search results.	Anthropic docs
`PerplexityBot`Perplexity	Surfaces and links websites in Perplexity search results. Not used to train AI models.	Perplexity docs
`Googlebot`Google	Google Search's crawler. A page must be indexed by Google Search, with a snippet allowed, to appear as a link in AI Overviews and AI Mode.	Google docs
`Bingbot`Microsoft	Bing's standard crawler, which handles most of Bing's crawling.	Microsoft docs
`Applebot`Apple	Powers search in Spotlight, Siri and Safari. Apple says the data may also help train its foundation models.	Apple docs
`DuckAssistBot`DuckDuckGo	Fetches pages in real time for DuckDuckGo's AI-assisted answers. Not used to train AI models.	DuckDuckGo docs
`Amzn-SearchBot`Amazon	Improves search in Amazon products and services. Not used for generative AI training.	Amazon docs
`MistralAI-Index`Mistral	Indexes pages for Mistral. Not used for training.	Mistral docs

Fetches when a person asks

Fetchers that load a page because someone asked an assistant about it.

Crawler	What it does	Source
`ChatGPT-User`OpenAI	Visits a page when a ChatGPT user's request needs it. Not used for automatic crawling. OpenAI says robots.txt rules may not apply.	OpenAI docs
`Claude-User`Anthropic	Fetches pages to support Claude users' requests.	Anthropic docs
`Perplexity-User`Perplexity	Fetches pages for actions a Perplexity user takes. Perplexity says it generally ignores robots.txt rules.	Perplexity docs
`meta-externalfetcher`Meta	Fetches individual links at a user's request. Meta says it may bypass robots.txt rules.	Meta docs
`Amzn-User`Amazon	Fetches live information from the web for Amazon users. Amazon says it may not follow all robots.txt directives.	Amazon docs
`MistralAI-User`Mistral	Fetches pages for users of Mistral's assistant. Not used for automatic crawling or training.	Mistral docs

AI training and datasets

Crawlers and controls that decide whether your content can be used to build AI models.

Crawler	What it does	Source
`GPTBot`OpenAI	Collects content that may be used to train OpenAI's generative AI foundation models.	OpenAI docs
`ClaudeBot`Anthropic	Collects web content that could contribute to training Anthropic's models.	Anthropic docs
`Google-Extended`Google	A robots.txt control, not a separate crawler. It decides whether content Google crawls may train Gemini models and ground answers in Gemini Apps and Vertex AI. It does not affect Google Search.	Google docs
`Applebot-Extended`Apple	A robots.txt control, not a crawler. It decides whether Applebot's data may train Apple's generative foundation models.	Apple docs
`meta-externalagent`Meta	Used to train foundation AI models or improve products by indexing content.	Meta docs
`Amazonbot`Amazon	Improves Amazon products and services. Amazon says it may be used to train Amazon AI models.	Amazon docs
`CCBot`Common Crawl	Builds Common Crawl's open repository of web crawl data.	Common Crawl docs
`MistralAI-Training`Mistral	Builds datasets for training Mistral's generative AI models.	Mistral docs

Search, user fetchers and training are three separate decisions

Most AI companies now run more than one crawler, and robots.txt treats each one on its own. Blocking one does not block the others.

Search and AI answer crawlers, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, decide whether an assistant can find your page and cite it. If you want to be recommended, keep these open.
User fetchers, such as ChatGPT-User and Perplexity-User, load a page when someone asks an assistant about it. Several companies say these may not follow robots.txt, because a person asked for the page.
Training crawlers and controls, such as GPTBot, ClaudeBot and Google-Extended, decide whether your content can be used to build models. OpenAI, Anthropic, Google and Apple document these separately from their search crawlers, so you can opt out of training and stay in AI search.

Example robots.txt rules

Stay open to AI search, opt out of model training

# Training crawlers and controls: opt out
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: MistralAI-Training
Disallow: /

# Everyone else, including search and AI answer crawlers
User-agent: *
Allow: /

Amazonbot is left out on purpose: Amazon uses it to improve its products as well as for training, so decide on it separately. Run this checker again after you publish the change.

Welcome every crawler

If you want every AI crawler, make sure no rule blocks them. A robots.txt with only User-agent: * and Allow: /, or no robots.txt at all, allows everything.

Avoid these mistakes

Do not disallow Googlebot or Bingbot to keep content out of AI. That removes the pages from Google Search or Bing too.
Search tags such as noindex and nosnippet change what appears in search, including Google's AI features. They are not training controls. For Google, the training control is Google-Extended.
Check robots.txt after redesigns, plugin updates and CDN changes, which often rewrite it.
Allow time for changes to apply. DuckDuckGo, for example, says a DuckAssistBot opt-out takes effect after 72 hours.

Why JavaScript matters

Many sites build their pages in the browser with JavaScript. A crawler that does not run JavaScript gets only the HTML the server sends, which can be an empty shell. In December 2024, Vercel and MERJ reported that none of the major AI crawlers they studied rendered JavaScript, including OpenAI's crawlers, ClaudeBot, Meta-ExternalAgent and PerplexityBot. Google's Gemini uses Googlebot's infrastructure, and Apple says Applebot may render pages in a browser.

This checker counts the words your server sends before any JavaScript runs. If the number is low, render the main content on the server.

What this checker cannot see

Firewalls and CDNs. A security rule can block AI crawlers that robots.txt allows. Google's guidance asks site owners to check that crawling is allowed in robots.txt and by any CDN or hosting setup.
Whether a crawler visited. Your server logs show which crawlers came and what they received.
Pages after JavaScript runs. Crawlers that do run JavaScript, such as Googlebot, may see more than the word count here.
Different answers for different visitors. We fetch the page as "VTurnU-AI-Crawler-Checker". If your site treats crawlers differently by name or address, they may get another answer.

Straight answers

AI crawler questions

Does blocking GPTBot remove my site from ChatGPT search?

OpenAI documents them separately. GPTBot collects content that may be used to train its models, while OAI-SearchBot finds pages for ChatGPT search. Each has its own robots.txt token, so you can block one and allow the other.

Does Google-Extended affect Google Search or AI Overviews?

Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. AI Overviews and AI Mode link to pages that are indexed in Google Search with a snippet, so they depend on Googlebot and on snippet controls such as nosnippet.

Do AI crawlers obey robots.txt?

The search and training crawlers listed here are documented with robots.txt controls. Several user fetchers are not: OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores them, and Meta and Amazon say their user fetchers may bypass them.

My robots.txt allows everything. Why can AI tools still not read my site?

Common causes are a firewall or CDN rule that blocks AI crawlers, main content that only appears after JavaScript runs, and noindex or nosnippet tags. This checker shows the last two. Your firewall settings and server logs show the first.

Do I need an llms.txt file?

No. llms.txt is a proposed convention, not a standard, and robots.txt is the control these companies document. The checker notes whether you have one.

How often should I check?

After any change to robots.txt, your CMS or plugins, your CDN or security settings, or a redesign. Companies also add and rename crawlers, and the reference list on this page shows the date it was last checked.

AI Crawler Checker