Are the AI crawlers allowed on your site? llms.txt & robots.txt

Before any GEO strategy matters, the engines have to be allowed in. Plenty of sites quietly block the AI crawlers in robots.txt — sometimes by default, sometimes from a copy-pasted config — and then wonder why they never appear in AI answers.

The crawlers that matter

• GPTBot & OAI-SearchBot — OpenAI / ChatGPT.
• PerplexityBot — Perplexity.
• ClaudeBot — Claude.
• Google-Extended — Google's AI features.

Check your robots.txt

Open yoursite.com/robots.txt and look for any 'Disallow: /' under these user-agents (or under '*'). If they're blocked, you've opted out of being recommended by those engines. Unblocking is usually a one-line change.

What llms.txt adds

An llms.txt file is a short, machine-readable summary of your business and key pages, placed at the root — a friendly map for AI crawlers. It doesn't replace good structure, but it makes you easier to read and cite. (Our audit checks both robots.txt and llms.txt for you.)