Your robots.txt allows every AI crawler. Yet when you ask ChatGPT or Perplexity about your company, the answer is thin, out of date, or about someone else.
"Allowed" is not the same as "readable". Between a crawler and your content there are six layers, and robots.txt is only the first. A firewall can refuse the request. A CDN setting can block it. The page can arrive with no text in it, because the text is added later by JavaScript the crawler never runs.
This guide is a fault-finding sequence. Test the six layers in order, mark each one Pass, Fail or Not sure, and the board builds your list of fixes. Each test takes a few minutes and most need nothing more than a terminal or your browser.
If robots.txt allows AI crawlers and assistants still cannot read your website, the block is in another layer. Test six, in order: the robots.txt rules, the firewall, the CDN's AI settings, the page's HTML without JavaScript, walls such as logins and consent screens, and the server log. The log is the only proof that a crawler arrived and what it received.
Three Kinds of AI Crawler, Three Different Consequences
Before testing anything, be clear about which crawlers you care about. AI companies run three kinds, and blocking one does not have the effect people expect on the others.
| Kind | What it does | Examples | If it cannot read you |
|---|---|---|---|
| Search and answer crawlers | Find and index pages so an assistant can show and link them | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot | You are not offered as a source in that assistant's search answers |
| Fetchers that act for a person | Load one page because someone asked about it | ChatGPT-User, Claude-User, Perplexity-User | The assistant tells the person it could not open your page |
| Training crawlers and controls | Collect content that may be used to train models | GPTBot, ClaudeBot, Google-Extended | Your content is left out of future training. Search answers are not affected |
The split matters because each vendor documents these separately. OpenAI lists OAI-SearchBot for ChatGPT search and GPTBot for training, each with its own robots.txt token. Anthropic does the same with Claude-SearchBot, Claude-User and ClaudeBot. Google's AI Overviews and AI Mode rely on Googlebot, while Google-Extended is a robots.txt token with no crawler of its own. Google says it controls the use of your content for Gemini training and grounding, and does not affect Google Search.
The fetchers follow different rules again. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person started the request. Perplexity says Perplexity-User generally ignores robots.txt for the same reason. Google's user-triggered fetchers generally ignore it too. So robots.txt is a reliable control for the first and third rows of the table, and a weak one for the second.
0 of 0 layers tested
Pick one page that matters, such as a service page or an article, and run every test on that page.
Halfway. Mark "Not sure" where you could not run the test; those are questions for your developer or host.
One layer to go. The server log is the one that settles it.
All six layers tested. Start with anything marked Fail, in the order shown under "Fix these first".
0 pass0 fail0 not sure
Fix these first
1. The robots.txt rules
Start here, because it is the cheapest test and the one every vendor documents. A rule can block a crawler by name, or block everything under a general rule that was left behind from a staging site.
How to test
Open yoursite.com/robots.txt in a browser, or run the AI Crawler Checker. Look for a Disallow: / under User-agent: *, and for rules that name the search crawlers in the table above.
Check every subdomain you publish on, such as a blog or a help centre. Anthropic's documentation says its opt-out applies per subdomain, and each subdomain has its own file.
What a failure looks like
- The search crawlers are disallowed, often by a rule meant only for training crawlers.
- The main site allows crawlers and the blog subdomain does not.
- The file was changed today. OpenAI and Perplexity both say a robots.txt change can take up to about 24 hours to take effect.
2. The firewall and bot rules
A web application firewall or a security plugin decides whether a request gets an answer at all. Many are set to challenge anything that looks automated, and an AI crawler looks automated because it is.
The test is to request the same page twice, once as a browser and once with a crawler's name in the user agent, and compare what comes back. Each command prints the status code and the size of the response.
# 1. As a normal browser
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0 Safari/537.36" https://www.example.com/your-page/
# 2. With a search crawler's name in the user agent
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" https://www.example.com/your-page/On Windows, use curl.exe and replace /dev/null with NUL. Repeat the second command with ClaudeBot and PerplexityBot in place of OAI-SearchBot.
How to read it
- 200 and a similar size for both: no blanket rule against that name.
- 403: refused by a rule.
- 429: rate limited.
- 503, or 200 with a much smaller size: probably a challenge page in place of your content.
Why this is only a first hint
Some firewalls check the address a request comes from, not only its name. Your test comes from your own address, so a firewall may refuse you as an impostor while letting the real crawler through, or the reverse.
A pass here rules out the crudest block. Layer 6 is the proof.
3. The CDN's AI settings
If your site sits behind a content delivery network, the network may have its own switch for AI crawlers, separate from your firewall rules and from the robots.txt file in your CMS. People are often surprised to find it, because nobody on the team chose the setting.
Cloudflare is the clearest example, and its defaults have moved. In July 2025 it announced that new domains would block AI crawlers by default. During 2026 it replaced that with separate policies for search, agents and training. Its documentation, updated in November 2026, says new domains now default to allowing all three. A domain added in between may have whatever was chosen, or defaulted, on the day it was set up.
How to test
Log in to the CDN and find the AI crawler or bot settings. Note what is set for search crawlers, agents and training.
Then fetch the live robots.txt and compare it with the file in your CMS:
curl -s https://www.example.com/robots.txt
What a failure looks like
- Search crawlers are set to Block, or a general "block AI bots" option is on.
- The live robots.txt has lines your CMS file does not. Cloudflare can write its policy into robots.txt for you.
- A bot protection mode is challenging every automated visitor. Perplexity's documentation notes that Cloudflare's Bot Fight Mode can still challenge its crawlers.
4. The page without JavaScript
This is the layer that catches modern websites. A page built with a JavaScript framework can send the browser an almost empty document and fill in the text afterwards. A person never notices. A crawler that does not run JavaScript receives the empty document and leaves.
The largest published study of this is by Vercel and MERJ, from December 2024. Across their network, none of the main AI crawlers they tracked ran JavaScript, including OpenAI's, Anthropic's and Perplexity's. Googlebot did. Google's own JavaScript guidance still recommends rendering on the server, because not every bot runs scripts.
A newer and smaller finding. In October 2026, small log tests reported OpenAI's search crawler running JavaScript some time after fetching the HTML. OpenAI has not confirmed it, and the tests cover very few sites. It does not change the advice: text that is in the HTML works for every crawler, today.
How to test
Fetch the page without a browser and count the lines that contain a sentence from the middle of your article:
curl -s https://www.example.com/your-article/ | grep -c "a sentence from the article"
No terminal? Open the page, press Ctrl and U to view the source, and search it for the sentence.
What a failure looks like
- The command prints 0, or the sentence is not in the source.
- The source is short and mostly script tags.
- Headings and menus are present, and the article body is not.
5. Walls in front of the content
A page can return a normal response with real HTML and still hide what matters. Anything a visitor has to click through, log in to or wait for is a wall to a crawler, which does none of those things.
What to look for
- Logins and gated pages. Specifications, pricing guidance or case studies behind a form.
- Consent screens that replace the page until someone accepts.
- Country blocks. Crawlers visit from data centres, often in another country.
- Content loaded on a click: tabs, accordions and "read more" links that fetch text only when pressed.
- Facts locked in PDFs or images.
How to test
Open the page in a private window with JavaScript switched off in the browser's settings. What you can read without clicking is roughly what a crawler gets.
Check the page for noindex and nosnippet too. Google says its AI features follow the same snippet controls as Search.
6. The server log
Every test so far is you, pretending to be a crawler. The log is the record of what happened when the real one came. It answers three questions: did it visit, which pages did it ask for, and what did it receive?
Ask your host for the access logs for the last 30 days, or use the crawler report in your CDN. Then count responses by status code for the AI crawlers:
grep -iE "OAI-SearchBot|ChatGPT-User|GPTBot|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User" access.log | awk '{print $9}' | sort | uniq -cThe output is a count beside each status code. This is an example of the shape, not data from a real site:
212 200
31 301
57 404
18 403How to read it
- No lines at all: the crawlers are not reaching the server. Go back to layers 1 to 3.
- 403 or 503: they arrive and are refused.
- Many 404s: they are asking for addresses that no longer exist. The Vercel study found that about a third of the requests from ChatGPT's and Claude's crawlers ended in a 404. Redirect old addresses.
- 200s on the pages that matter: access is working.
Check that the visitor is genuine
Anyone can send a request with a crawler's name. OpenAI, Anthropic and Perplexity each publish the address ranges their crawlers use, linked from the documentation pages above. Compare the addresses in your log with those lists before you trust a line.
Two Diagnoses From Our Own Work
Both of these were layer 4 failures. Both sites have a robots.txt that allows AI crawlers, which we checked on 14 November 2026. On both, the articles were unreadable to a crawler that does not run JavaScript.
- FlowEazy. The blog loaded its articles through JavaScript, so AI crawlers received a page without the article. The fix is to render article pages on the server. It is built and, at the time of writing, waiting for the client's final server upload. The FlowEazy case study has the background.
- Cafy. Blog articles stayed invisible until JavaScript ran. We documented the fix for the client's developer, who shipped it. When we repeated the layer 4 test on 14 November 2026, an article returned its full text, about 2,000 words, in the raw HTML. See the Cafy case study.
Neither problem showed up in a robots.txt check, a rankings report or a speed score. Our article on what a PageSpeed score cannot see looks at the same blind spot from the other side.
From Symptom to Layer
If you cannot test everything today, start from what you have noticed. Each symptom points to the layers most likely to be at fault.
| What you notice | Look first at | Why |
|---|---|---|
| You rank in Google, and ChatGPT or Perplexity never cite you | Layers 1, 2 and 3 | Googlebot is allowed through, and the other crawlers are refused by a bot rule |
| Assistants cite your home page and never your articles | Layer 4 | The home page is plain HTML, and the articles are filled in by JavaScript |
| Assistants describe you with old information | Layers 1 and 6 | The search crawler is blocked or has not visited, so the answer leans on older data |
| Citations stopped suddenly | Layers 2 and 3 | A firewall rule, a security plugin or a CDN default changed |
| An assistant says it could not open your page | Layers 2, 3 and 5 | The fetcher that acts for a person was challenged or met a wall |
| Specifications or certificates are missing from answers | Layers 4 and 5 | The facts are in a PDF, an image or a tab that loads on a click |
Who Fixes What
Access problems last because each layer belongs to someone different, and none of them is asked the whole question. This table is the message to send.
| Layer | Who can change it | What to ask for |
|---|---|---|
| 1. robots.txt | Marketing, with the developer | A written policy: which crawlers are allowed, which are not, and why |
| 2. Firewall | Host or security team | Verified AI search crawlers allowed through, matched by the operators' published address ranges |
| 3. CDN settings | Whoever holds the CDN account | A review of the AI crawler policies and the bot protection mode |
| 4. Rendering | Developer | Main text, headings and links present in the HTML the server sends |
| 5. Walls | Developer and marketing | Public pages readable without a login, a consent click or a second request |
| 6. Logs | Host or developer | Thirty days of access logs, or a crawler report from the CDN, every month |
Whether to allow training crawlers is a separate decision from all of this, and it belongs to the business, not to the developer. You can allow search and refuse training. The vendors document the two separately so that you can.
Three Things Not to Do
- Do not serve crawlers a different page from the one visitors see. Search engines treat that as cloaking. Fix the page for everyone.
- Do not block every AI crawler to "protect content" without knowing the cost. Blocking the search crawlers removes you from the answers your buyers are reading. Decide per kind of crawler.
- Do not treat one passed test as permanent. A plugin update, a new firewall rule or a changed CDN default can close a layer overnight. Repeat layers 2, 3 and 6 every quarter.
Access is also only the first condition. A crawler that can read your page still has to find something worth quoting. Our guide to measuring AI search visibility covers how to track what happens next, and how AI agents evaluate a B2B website covers the agents that click and type, which face a different set of obstacles.