If your business is invisible in ChatGPT, the instinctive assumption is that you lost some kind of ranking contest. Usually you didn’t. Very often you were never in the running, because a crawler was blocked at the door — sometimes deliberately, frequently by accident, and almost always by a robots.txt file written before most of these crawlers existed.
This is a crawlability problem with a technical answer, and it is genuinely checkable. You can establish where you stand in about two minutes.
The mistake almost everyone makes
There is no such thing as “the ChatGPT crawler.”
OpenAI operates several distinct crawlers, each with a different job, each independently controllable in robots.txt. Blocking one does not block the others, and — critically — they do not do the same thing at all:
- GPTBot is used for model training.
- OAI-SearchBot is what surfaces your site inside ChatGPT search.
- ChatGPT-User fetches a page when a user explicitly asks ChatGPT to visit it.
- OAI-AdsBot validates landing pages submitted for advertising.
OpenAI’s own bot documentation is unambiguous about the consequence: sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers.
Read that again, because the business implication is large. Declining to have your content train a model and remaining visible in ChatGPT are two different decisions, and most organisations want to make them differently. Plenty of businesses have a considered, defensible position that their content should not be used for training. Almost none of them intend to become invisible to buyers researching in ChatGPT. Yet a single overly broad Disallow does exactly that.
The same split exists elsewhere. Anthropic runs ClaudeBot for training, Claude-SearchBot for search relevance, and Claude-User for user-initiated fetches, all documented in Anthropic’s crawler guidance. Perplexity runs PerplexityBot, which surfaces and links sites in its results and is explicitly not used for training, alongside Perplexity-User for user-initiated requests — see Perplexity’s crawler documentation.
Three vendors. Nine crawlers. Three genuinely different purposes — and every one of them sits underneath whatever GEO work you have commissioned, because none of it matters if the crawler never arrives. One robots.txt file that, on most sites, treats them as if they were one thing.
The silent override that breaks well-intentioned configurations
This is the part that catches careful people, and it is worth understanding properly because the failure is completely invisible.
Under the robots.txt specification, a crawler obeys only its single most specific matching group. It does not combine your named group with your wildcard group. It picks one and ignores the rest.
So consider a site that has sensibly set up a wildcard group with real rules in it, then decides to signal openness to AI crawlers by adding this:
User-agent: *
Disallow: /api/
Content-Signal: search=yes, ai-input=yes, ai-train=no
User-agent: GPTBot
Allow: /The intent is obvious and reasonable. The effect is not. GPTBot now matches its own named group, so it stops reading the wildcard group entirely — including the Disallow: /api/ rule and the entire content-signal declaration. Every rule the site owner thought applied to GPTBot has been silently switched off by the act of welcoming it.
Nothing errors. Nothing warns you. A validator will call the file valid, because it is valid — it simply does not do what the author meant.
We know this pattern well because we found it on our own site. An audit of doublebarril.com in August 2026 flagged exactly this: per-agent groups containing only Allow: /, which meant the named AI crawlers never saw the global content-signal line at all. The fix was to collapse everything back to a single wildcard group, and the reasoning is written into the file itself so the bug cannot quietly reappear the next time someone tidies it up. You can read our current robots.txt and see both the configuration and the comment explaining why it is shaped that way.
If you are going to sell technical rigour, your own implementation should be the first thing you are willing to show.
What each crawler actually does
| Crawler | Vendor | What it does | Blocking it means |
|---|---|---|---|
GPTBot | OpenAI | Model training | Your content is not used for training. Does not affect ChatGPT search visibility |
OAI-SearchBot | OpenAI | ChatGPT search inclusion | You will not appear in ChatGPT search answers |
ChatGPT-User | OpenAI | User-initiated page fetch | A user asking ChatGPT to read your page may fail. Not used for search inclusion |
OAI-AdsBot | OpenAI | Ad landing-page validation | Affects advertising submissions only |
ClaudeBot | Anthropic | Model training | Content not used for training |
Claude-SearchBot | Anthropic | Search relevance and accuracy | Reduced accuracy of how Claude represents you |
Claude-User | Anthropic | User-initiated fetch | A user-requested read of your page may fail |
PerplexityBot | Perplexity | Surfacing and linking your site in results | You will not be surfaced or linked in Perplexity |
Perplexity-User | Perplexity | User-initiated fetch | Perplexity documents that this one generally ignores robots.txt |
Googlebot | Search, including AI Overviews and AI Mode | You leave Google Search entirely | |
Google-Extended | Gemini app and Vertex AI training | Does not remove you from AI Overviews or AI Mode |
That last row is worth dwelling on, because it is widely misreported. Google-Extended is a training control. It is not an AI Overviews opt-out. Whether you appear in AI Overviews or AI Mode is governed by whether the page is indexed and by ordinary snippet controls — nosnippet, data-nosnippet, max-snippet — as set out in Google’s documentation on AI features. There is no separate AI opt-out lever, and there is no separate AI opt-in lever either.
What Google says you do not need to do
While we are here, it is worth being straight about the other half of this conversation, because a lot of money is currently being spent on the opposite advice.
Google’s AI optimisation guide states there are no additional requirements to appear in AI Overviews or AI Mode, and no special optimisations necessary. It says explicitly that you do not need to create machine-readable files or AI text files, that there is no special structured data you need to add for AI features, and that “optimizing for generative AI search is optimizing for the search experience, and thus still SEO.”
So the honest position is this: crawler access is a real, technical, checkable lever that materially determines whether an AI system can see you at all. A great deal of what is sold alongside it is not. We would rather tell you which is which than bill you for both.
How to check your own site in two minutes
- Open
yourdomain.com/robots.txt. Read it as a crawler would, not as a human does. - Count your groups. Every
User-agent:line starts a new group. If any AI crawler has its own named group, that crawler is reading only that group — check what it is missing from your wildcard rules. - Look for over-broad blocks. A
Disallow: /under any AI user-agent is a deliberate-looking decision that is frequently accidental, inherited from a template, or left over from a staging configuration that shipped to production. - Separate the two decisions. Decide training access and search visibility independently, per vendor. They are not the same choice and should not be made with one rule.
- Then verify it in your server logs. This is the step almost nobody does, and it is the only one that produces evidence. Your access logs record every request by user agent. OpenAI, Anthropic and Perplexity all publish verified IP ranges, so you can confirm that a request claiming to be their crawler genuinely came from them. Until you have looked at logs, you have a
robots.txtyou believe works.
A note on timing: none of this is instant. Perplexity documents that robots.txt changes can take up to 24 hours to reflect. Make the change, then verify later — do not assume.
What we cannot tell you
Consistent with how we report on AI visibility generally, here are the limits of this analysis, stated up front rather than discovered later:
- Crawler access is necessary, not sufficient. Being reachable does not mean being cited. It means being eligible. Anyone promising citations in exchange for a robots.txt edit is overselling.
- Perplexity publishes no citation methodology. Any claim about how it chooses sources is an assertion, not a citation.
- Google Search Console’s generative AI report gives impressions only — no clicks, no click-through rate, no position — and those impressions are already included in your existing web search totals rather than added to them. Any report that adds them to your organic traffic is double-counting.
Content-Signaldeclarations are a stated preference, not an enforcement mechanism. They express your intent. They do not compel anyone.- Access does not fix representation. If an engine reaches you but describes you wrongly, that is an entity problem, not a crawler one. We keep the current evidence on AI search behaviour separate from what we can demonstrate on your own estate.
The short version
Being invisible in AI answers is often not a content problem, a schema problem, or a strategy problem. It is a robots.txt file that treats nine different crawlers as if they were one, written before the distinction existed, and never verified against a server log.
That is a cheap thing to check and an expensive thing to leave alone.
If you want this done properly — every AI crawler’s access verified against your actual server logs, with the training and search decisions separated and documented per vendor — that is the first deliverable in our Visibility Diagnostic. If you would rather start by seeing where you stand across the answer engines themselves, the AI Visibility Checker walks you through it in a few minutes.