# llms.txt and structured data: what AI assistants actually use Source: https://botofmouth.com/guides/llms-txt-and-structured-data/ 11 November 2026 If you build websites for small businesses, you've been asked some version of this: “Can you add the file that makes ChatGPT recommend us?” The usual suspects are llms.txt and schema.org structured data. Both are cheap to ship. Here is what the public record says each one does, and the less glamorous checks that decide whether an assistant can read a site at all. ## The short version - **llms.txt:** Google Search ignores it, fetches are rare, and no study has measured an effect on being cited. Ship it for docs and developer tools; it's optional elsewhere. - **JSON-LD:** not required for AI answers, and controlled tests show no clear lift. Models do read it, as plain text. Ship it, if it matches the visible page. - **Crawler access:** a blocked search crawler means no appearance in that assistant's search answers. Check this first. - **Server-rendered text:** most AI crawlers don't run JavaScript. Check this second. ## llms.txt The [spec](https://llmstxt.org/) is small: a Markdown file at /llms.txt. The H1 with the site name is the only required part. After it come a blockquote summary and H2 sections of links. ``` # Harbour Street Dental > A dental practice in Whitby. General and cosmetic dentistry, > with extra time for anyone who's nervous. ## Pages - [Treatments](https://example.com/treatments.md): every treatment we offer - [Opening hours and contact](https://example.com/contact.md) ## Optional - [Meet the team](https://example.com/team.md) ``` Who reads it? - **Google doesn't.** Its guide says such files [“will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them”](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide). John Mueller has compared llms.txt to [the keywords meta tag](https://www.searchenginejournal.com/google-says-llms-txt-comparable-to-keywords-meta-tag/544804/). - **Fetches are rare.** Ahrefs looked at 137,000 domains in 2026. 28% publish the file, and [“97% of those files received zero traffic in May 2026”](https://ahrefs.com/blog/llmstxt-study/). Among the fetches that did come from named AI tools, GPTBot was first and Claude Code second. - **No measured effect on citations.** SE Ranking studied about 300,000 domains and found [no effect on how often a domain is cited](https://seranking.com/blog/llms-txt/). - **Chrome treats it as optional.** Lighthouse has an llms.txt audit in its agentic-browsing category, and a missing file is reported as [“Not Applicable”](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt). That second-place fetcher is the real use case: coding agents pull llms.txt to read documentation. A related idea is worth more of your time: serving Markdown to agents that ask for it. Cloudflare's [Markdown for Agents](https://blog.cloudflare.com/markdown-for-agents/) answers an Accept: text/markdown request with a Markdown version of the page. **Verdict:** for a local business site, llms.txt is a five-minute nicety. We publish one on our own site because it costs nothing. Don't sell it as the thing that gets a business recommended. ## Structured data Google is just as direct: [“Structured data isn't required for generative AI search, and there's no special schema.org markup you need to add”](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide). Its one rule for AI features is consistency: structured data should [match the visible text on the page](https://developers.google.com/search/docs/appearance/ai-features). Microsoft is warmer, calling schema [“a type of code that helps search engines and AI systems understand your content”](https://about.ads.microsoft.com/en/blog/post/october-2025/optimizing-your-content-for-inclusion-in-ai-search-answers), while adding that nothing guarantees selection. Two experiments are more useful than either statement: - **Ahrefs** added JSON-LD to 1,885 pages and watched for citations. The changes were small and went both ways: [AI Overviews −4.6%, AI Mode +2.4%, ChatGPT +2.2%](https://www.searchenginejournal.com/schema-markup-didnt-move-ai-citations-in-ahrefs-test/582823/). No clear effect. - **Mark Williams-Cook** put a made-up address into invalid JSON-LD and nowhere else on the page. [ChatGPT and Perplexity returned it](https://www.seroundtable.com/chatgpt-perplexity-structured-data-text-40862.html). The model wasn't parsing schema; it was reading the HTML. Put together, JSON-LD stops being magic. It's one more place where the facts sit as text: useful when they're right, harmful when they've drifted from the page. So ship the basics, and keep them true: ``` ``` Use the [most specific LocalBusiness subtype](https://developers.google.com/search/docs/appearance/structured-data/local-business) that fits, as Google advises. And don't count on FAQ markup for display: Google's FAQ rich results [stopped appearing on 7 May 2026](https://developers.google.com/search/updates). Questions and answers as visible text are still worth having. ## What decides whether a site is read at all **1. A firewall that blocks the crawlers you invited.** Google's checklist asks that crawling is allowed [“in robots.txt, and by any CDN or hosting infrastructure”](https://developers.google.com/search/docs/appearance/ai-features), and since July 2025 Cloudflare has [blocked AI crawlers by default](https://www.cloudflare.com/press/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/), asking every new domain whether to let them in. OpenAI is clear about the stakes: sites that opt out of OAI-SearchBot [“will not be shown in ChatGPT search answers”](https://platform.openai.com/docs/bots). A quick test: ``` for ua in OAI-SearchBot GPTBot ClaudeBot Claude-SearchBot PerplexityBot; do code=$(curl -s -o /dev/null -w '%{http_code}' \ -A "Mozilla/5.0 (compatible; $ua/1.0)" https://example.com/) echo "$ua $code" done ``` This only catches rules keyed on the user agent. A 200 doesn't prove the real crawler gets in, because some firewalls also check IP ranges. A 403 is a finding: confirm it in the server or CDN logs. **2. Pages that are empty without JavaScript.** Vercel and MERJ found that [none of the major AI crawlers rendered JavaScript](https://vercel.com/blog/the-rise-of-the-ai-crawler), apart from Gemini, which uses Googlebot. Count the words in the raw HTML: ``` curl -s https://example.com/menu/ | sed 's/<[^>]*>/ /g' | wc -w ``` If the number is a few dozen and the page looks full in a browser, the content arrives later from an API, and most assistants never see it. **3. Facts that disagree.** A phone number on the site that differs from the one on Google Maps. Two addresses. A Maps category that doesn't match what the business sells. OpenAI warns that [“search results and citations can be incomplete, outdated, or incorrect”](https://help.openai.com/en/articles/9237897-chatgpt-search), and conflicting sources give it more ways to be wrong. The fix isn't a file. It's making the sources agree. ## An order of work 1. Search crawlers can fetch the site: robots.txt, firewall, logs. 2. Key facts are in the server-rendered HTML. 3. Facts match across the site, the Google Business Profile and the main directories. 4. JSON-LD repeats those same facts. 5. llms.txt and Markdown versions, if you have ten minutes left. Steps 1 to 3 are where the problems usually are. Steps 4 and 5 are where people start. For more on step 2, see [Server-render the facts](https://botofmouth.com/guides/server-render-the-facts/). --- Want this as a to-do list for your own business? Start with the free check at [botofmouth.com](https://app.botofmouth.com/start). Also published on [Hashnode](https://botofmouth.hashnode.dev/llms-txt-and-structured-data-what-ai-assistants-actually-use).