Making your website readable for AI

Updated 27 September 2026 · PDF version

Let the right crawlers in

Each provider runs several crawlers with different jobs. You need to tell apart the ones used for search (and therefore for citing you) from the ones used to train models.

• OAI-SearchBot (OpenAI): surfaces websites in ChatGPT search. OpenAI recommends allowing it.
• GPTBot (OpenAI): collects content for training. According to OpenAI, blocking it doesn’t stop you appearing in ChatGPT search.
• Claude-SearchBot and Claude-User (Anthropic): search and on-demand fetches. Anthropic warns that blocking them may reduce your visibility. ClaudeBot, on the other hand, is used for training.
• PerplexityBot: surfaces and links websites in Perplexity.
• Google-Extended: a token that controls whether your content is used to train Gemini and for grounding in Gemini apps. Google states that it has no effect on inclusion or ranking in Google Search.

Pages an AI understands first time

• One page per service, with an explicit title: “Accountancy for freelancers and consultants in Birmingham” rather than “Our solutions”.
• Key facts written as text, not only in images: address, area covered, opening hours, price range, lead times, accreditations.
• A detailed “About” page: year founded, team, specialisms, public references you’re allowed to mention.
• Content that is visible without relying entirely on JavaScript, because not every crawler runs scripts in the same way.
• The same information everywhere: same name, address and phone number on your site, your listings and your directory entries.

Structured data and FAQs

Structured data describes your page in a standard format (the schema.org vocabulary). Google recommends the JSON-LD format. For a small business, the useful types are Organization or LocalBusiness (with the sameAs property pointing to your official profiles), Service, Product and Review where relevant.

Let’s be honest: Google says no special markup is needed to appear in its AI features. Structured data is not a magic formula. It still helps remove any ambiguity about who you are, and it costs next to nothing to set up.

A good FAQ uses your customers’ real questions, with short, factual answers, backed by figures where possible. That’s exactly the format assistants like to reuse.

What about llms.txt?

llms.txt is a convention proposed in September 2024 by Jeremy Howard: a Markdown file placed at the root of your site that summarises it and lists its key pages for language models. It isn’t an official standard, and whether the major assistants use it remains unclear. Google says that no special “text for AI” file is needed for its AI features.

Our advice: create one if it takes you an hour, because it forces you to spell out what you offer. But never put it ahead of crawler access, page quality and reviews.

Technical checklist

• Robots.txt doesn’t block OAI-SearchBot, Claude-SearchBot, PerplexityBot or Googlebot
• Your firewall or CDN doesn’t block AI search crawlers
• Your important pages are indexed in Google and Bing
• One dedicated page per service and per main area served
• Name, address and phone number identical on every page
• JSON-LD Organization or LocalBusiness markup with sameAs
• An FAQ built from real customer questions
• Main content readable without running JavaScript
• An llms.txt file (optional, once everything else is done)