Server-render the facts: a small-business site AI crawlers can read

Each vendor runs separate user agents for search, for user-triggered fetches and for model training, and documents them. The search ones are the ones that decide whether a site can appear in answers.

• OpenAI: OAI-SearchBot is “used to surface websites in search results in ChatGPT's search features”. GPTBot is for training. ChatGPT-User visits a page when a user asks, and “robots.txt rules may not apply” to it.
• Anthropic: Claude-SearchBot indexes for search, Claude-User fetches at a user's request, and ClaudeBot collects training data. Disabling Claude-SearchBot “may reduce your site's visibility and accuracy in user search results.”
• Perplexity: PerplexityBot “is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models.” Perplexity-User handles user requests.
• Google: AI Overviews and AI Mode use the normal Search index. A page needs to be “indexed and eligible to be shown in Google Search with a snippet”, with “no additional technical requirements.” The separate Google-Extended token controls Gemini training and grounding, and “does not impact a site's inclusion in Google Search”.

Vercel and MERJ watched AI crawler traffic across Vercel's network and found that none of the major AI crawlers currently render JavaScript: not OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot or PerplexityBot. ChatGPT and Claude did fetch JavaScript files (11.50% and 23.84% of requests) but didn't run them. Only Gemini, through Googlebot, and AppleBot rendered pages. Their advice: “any important content should be server-rendered.”

Google says the same from its side: “server-side or pre-rendering is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript.”

For a small business, the facts that must be in that first HTML response are few: the name, what the business does, the town or area it serves, the address, the phone number, the opening hours and the main services. Plain elements are enough:

<h1>Harbour Street Bakery</h1>
<p>Sourdough, pastries and coffee in Whitby, North Yorkshire.</p>
<p>Open Tuesday to Sunday, 7:00 to 15:00. Closed Monday.</p>
<address>
  12 Harbour Street, Whitby YO21 0AA<br>
  <a href="tel:+441947000000">01947 000000</a>
</address>

Static generation, server rendering or a plain HTML file all work. Rendering in the browser is fine for the extras: a chat widget, a live feed, a map. Then test what a crawler gets, before any JavaScript runs:

curl -s https://example.com/ | grep -c "01947 000000"
curl -s https://example.com/ | sed 's/<[^>]*>/ /g' | wc -w

If the phone number isn't found, or the word count is a few dozen for a page that looks full in a browser, the content arrives later from an API, and most assistants never see it. Text inside images and PDFs has the same problem: Google's checklist asks that “important content is available in textual form.”

The file must sit at “/robots.txt” (all lowercase) in the top-level path, served as plain text, and Anthropic notes it is needed for every subdomain. OpenAI supports allowing OAI-SearchBot while disallowing GPTBot, so search and training are separate choices. A site that wants to appear in answers but stay out of training could use:

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Googlebot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

That last group is a choice, not a requirement; leaving it out allows everything. Three cautions:

• robots.txt is not the only gate. Google asks that crawling be allowed “in robots.txt, and by any CDN or hosting infrastructure”. Cloudflare has blocked AI crawlers by default since July 2025 and asks every new domain whether to let them in. Check the firewall and bot settings too, and the server logs for 403s.
• Think twice about Google-Extended. It controls training and also grounding in Gemini Apps, so blocking it may keep the site out of some Gemini answers even though Google Search is unaffected.
• Changes take time. OpenAI says it “can take ~24 hours” after a robots.txt update for its search systems to adjust.

And robots.txt rules “are not a form of access authorization”. They are a request that well-behaved crawlers honour, not a lock.

Google says a site of about 500 pages or fewer that is well linked internally might not need a sitemap. It costs little, though, and the AI crawlers have a reason to want one: in the Vercel data, ChatGPT spent 34.82% of its fetches on 404 pages and Claude 34.16%, against 8.22% for Googlebot. Vercel's advice is to keep proper redirects, up-to-date sitemaps and consistent URL patterns.

List the sitemap in robots.txt with a Sitemap: line, which Google says can go “anywhere in your robots.txt file”. When a page moves, redirect the old URL instead of letting it 404.

Structured data is not a shortcut. Google says it “isn't required for generative AI search, and there's no special schema.org markup you need to add”, and that it should match the visible text on the page. It is still worth a few lines: use the most specific LocalBusiness subtype that fits, with the same name, address, phone and hours as the HTML above.

Ship it in the server response, like everything else. Vercel notes that content “included in the initial HTML response, like JSON data” can be read by crawlers that don't run JavaScript. JSON-LD injected by a tag manager after load is visible to Google, and possibly to no one else.

1. The name, area, address, phone, hours and services are in the HTML the server sends.
2. robots.txt allows OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot, on every subdomain.
3. The CDN, firewall and host don't block them either. Check the logs.
4. A sitemap is listed in robots.txt, and moved pages redirect.
5. JSON-LD repeats the same facts, in the first response.
6. The same facts appear on the Google Business Profile and the main directories.

None of this guarantees a mention in any assistant's answer. It makes sure that, when a crawler comes looking, it finds the facts.