# Server-render the facts: a small-business site AI crawlers can read Source: https://botofmouth.com/guides/server-render-the-facts/ 12 November 2026 Before an AI assistant can name a business, one of its crawlers has to fetch a page and find the facts in it. For most of those crawlers, the page is the HTML your server sends, not what the browser builds afterwards. This is a short, vendor-sourced checklist for making that HTML count. ## Know who is fetching Each vendor runs separate user agents for search, for user-triggered fetches and for model training, and documents them. The search ones are the ones that decide whether a site can appear in answers. - **OpenAI:** [OAI-SearchBot](https://platform.openai.com/docs/bots) is “used to surface websites in search results in ChatGPT's search features”. GPTBot is for training. ChatGPT-User visits a page when a user asks, and “robots.txt rules may not apply” to it. - **Anthropic:** [Claude-SearchBot](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) indexes for search, Claude-User fetches at a user's request, and ClaudeBot collects training data. Disabling Claude-SearchBot “may reduce your site's visibility and accuracy in user search results.” - **Perplexity:** [PerplexityBot](https://docs.perplexity.ai/guides/bots) “is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models.” Perplexity-User handles user requests. - **Google:** AI Overviews and AI Mode use the normal Search index. A page needs to be [“indexed and eligible to be shown in Google Search with a snippet”](https://developers.google.com/search/docs/appearance/ai-features), with “no additional technical requirements.” The separate Google-Extended token controls Gemini training and grounding, and [“does not impact a site's inclusion in Google Search”](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers). ## Put the facts in the first response Vercel and MERJ watched AI crawler traffic across Vercel's network and found that [none of the major AI crawlers currently render JavaScript](https://vercel.com/blog/the-rise-of-the-ai-crawler): not OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot or PerplexityBot. ChatGPT and Claude did fetch JavaScript files (11.50% and 23.84% of requests) but didn't run them. Only Gemini, through Googlebot, and AppleBot rendered pages. Their advice: “any important content should be server-rendered.” Google says the same from its side: [“server-side or pre-rendering is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript.”](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) For a small business, the facts that must be in that first HTML response are few: the name, what the business does, the town or area it serves, the address, the phone number, the opening hours and the main services. Plain elements are enough: ```

Harbour Street Bakery

Sourdough, pastries and coffee in Whitby, North Yorkshire.

Open Tuesday to Sunday, 7:00 to 15:00. Closed Monday.

12 Harbour Street, Whitby YO21 0AA
01947 000000
``` Static generation, server rendering or a plain HTML file all work. Rendering in the browser is fine for the extras: a chat widget, a live feed, a map. Then test what a crawler gets, before any JavaScript runs: ``` curl -s https://example.com/ | grep -c "01947 000000" curl -s https://example.com/ | sed 's/<[^>]*>/ /g' | wc -w ``` If the phone number isn't found, or the word count is a few dozen for a page that looks full in a browser, the content arrives later from an API, and most assistants never see it. Text inside images and PDFs has the same problem: Google's checklist asks that [“important content is available in textual form.”](https://developers.google.com/search/docs/appearance/ai-features) ## robots.txt: let the search crawlers in The file must sit at [“/robots.txt” (all lowercase) in the top-level path](https://www.rfc-editor.org/rfc/rfc9309), served as plain text, and Anthropic notes it is needed [for every subdomain](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler). OpenAI supports allowing OAI-SearchBot while disallowing GPTBot, so search and training are separate choices. A site that wants to appear in answers but stay out of training could use: ``` User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Googlebot Allow: / User-agent: GPTBot User-agent: ClaudeBot Disallow: / Sitemap: https://example.com/sitemap.xml ``` That last group is a choice, not a requirement; leaving it out allows everything. Three cautions: - **robots.txt is not the only gate.** Google asks that crawling be allowed [“in robots.txt, and by any CDN or hosting infrastructure”](https://developers.google.com/search/docs/appearance/ai-features). Cloudflare has [blocked AI crawlers by default](https://www.cloudflare.com/press/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/) since July 2025 and asks every new domain whether to let them in. Check the firewall and bot settings too, and the server logs for 403s. - **Think twice about Google-Extended.** It controls training and also [grounding in Gemini Apps](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), so blocking it may keep the site out of some Gemini answers even though Google Search is unaffected. - **Changes take time.** OpenAI says it [“can take ~24 hours”](https://platform.openai.com/docs/bots) after a robots.txt update for its search systems to adjust. And robots.txt rules [“are not a form of access authorization”](https://www.rfc-editor.org/rfc/rfc9309). They are a request that well-behaved crawlers honour, not a lock. ## A sitemap, and URLs that don't break Google says a site of [about 500 pages or fewer](https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview) that is well linked internally might not need a sitemap. It costs little, though, and the AI crawlers have a reason to want one: in the Vercel data, ChatGPT spent [34.82% of its fetches on 404 pages](https://vercel.com/blog/the-rise-of-the-ai-crawler) and Claude 34.16%, against 8.22% for Googlebot. Vercel's advice is to keep proper redirects, up-to-date sitemaps and consistent URL patterns. List the sitemap in robots.txt with a Sitemap: line, which Google says can go [“anywhere in your robots.txt file”](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap). When a page moves, redirect the old URL instead of letting it 404. ## Structured data: the same facts, once more Structured data is not a shortcut. Google says it [“isn't required for generative AI search, and there's no special schema.org markup you need to add”](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide), and that it should [match the visible text on the page](https://developers.google.com/search/docs/appearance/ai-features). It is still worth a few lines: use the [most specific LocalBusiness subtype](https://developers.google.com/search/docs/appearance/structured-data/local-business) that fits, with the same name, address, phone and hours as the HTML above. Ship it in the server response, like everything else. Vercel notes that content [“included in the initial HTML response, like JSON data”](https://vercel.com/blog/the-rise-of-the-ai-crawler) can be read by crawlers that don't run JavaScript. JSON-LD injected by a tag manager after load is visible to Google, and possibly to no one else. ## The checklist 1. The name, area, address, phone, hours and services are in the HTML the server sends. 2. robots.txt allows OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot, on every subdomain. 3. The CDN, firewall and host don't block them either. Check the logs. 4. A sitemap is listed in robots.txt, and moved pages redirect. 5. JSON-LD repeats the same facts, in the first response. 6. The same facts appear on the Google Business Profile and the main directories. None of this guarantees a mention in any assistant's answer. It makes sure that, when a crawler comes looking, it finds the facts. --- Want this as a to-do list for your own business? Start with the free check at [botofmouth.com](https://app.botofmouth.com/start).