Are the AI crawlers allowed on your site? llms.txt & robots.txt

Before any GEO strategy matters, the engines have to be allowed in. This is a rare failure — almost no small business blocks these agents, which is why our own scan weights crawler access at 2 of 40.5 points — but it is binary, so it is worth two minutes to rule out. The bigger trap is assuming every bot carrying an AI company's name controls the same thing. They do not.

Which crawler controls what

• OAI-SearchBot — OpenAI's agent for ChatGPT search. This is the one that governs whether you can be cited in ChatGPT's web-grounded answers.
• GPTBot — OpenAI's training crawler. Blocking it opts you out of model training, not out of ChatGPT search.
• ChatGPT-User — fetches a page because a person asked for it in the moment; OpenAI notes robots.txt rules may not apply to it.
• Claude-SearchBot — Anthropic's search agent, the counterpart to OAI-SearchBot. ClaudeBot is the training crawler and Claude-User is the user fetch.
• PerplexityBot — Perplexity's index crawler.
• Googlebot — the crawler behind AI Overviews and AI Mode, which are generated from Google's ordinary Search index.
• Google-Extended — a robots.txt token rather than a user agent. It governs Gemini training and grounding, and Google states it does not affect a site's inclusion in Google Search.

Check your robots.txt

Open yoursite.com/robots.txt and look for any 'Disallow: /' under these user-agents, or under '*'. What costs you answers is blocking a search agent — OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot — and even then OpenAI notes that opted-out sites can still appear as navigational links. Blocking a training crawler costs you nothing in search. Remember too that robots.txt is not the only gate: a CDN or WAF bot rule can block the same agents without ever appearing in the file.

What llms.txt adds

An llms.txt file is a short, machine-readable summary of your business and key pages, placed at the root. No answer engine has confirmed using it, and Google has said its own systems do not — so treat it as a cheap, harmless statement of who you are, not as a visibility lever. We publish one ourselves, mainly as entity and disambiguation text. Our free scan reports whether you have robots.txt and llms.txt; it does not claim the second earns you anything.

Where we apply this

The trades and cities our research covers, and the service behind it.

More in GEO

Find out what an answer engine can actually read about you.

The free scan reads your site the way an answer engine's crawler does and shows you the gaps to close first. It does not query ChatGPT, Perplexity or Google — that check is the human deep-dive.