llms.txt and structured data: what AI assistants actually use

• llms.txt: Google Search ignores it, fetches are rare, and no study has measured an effect on being cited. Ship it for docs and developer tools; it's optional elsewhere.
• JSON-LD: not required for AI answers, and controlled tests show no clear lift. Models do read it, as plain text. Ship it, if it matches the visible page.
• Crawler access: a blocked search crawler means no appearance in that assistant's search answers. Check this first.
• Server-rendered text: most AI crawlers don't run JavaScript. Check this second.

The spec is small: a Markdown file at /llms.txt. The H1 with the site name is the only required part. After it come a blockquote summary and H2 sections of links.

# Harbour Street Dental

> A dental practice in Whitby. General and cosmetic dentistry,
> with extra time for anyone who's nervous.

## Pages
- [Treatments](https://example.com/treatments.md): every treatment we offer
- [Opening hours and contact](https://example.com/contact.md)

## Optional
- [Meet the team](https://example.com/team.md)

Who reads it?

• Google doesn't. Its guide says such files “will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them”. John Mueller has compared llms.txt to the keywords meta tag.
• Fetches are rare. Ahrefs looked at 137,000 domains in 2026. 28% publish the file, and “97% of those files received zero traffic in May 2026”. Among the fetches that did come from named AI tools, GPTBot was first and Claude Code second.
• No measured effect on citations. SE Ranking studied about 300,000 domains and found no effect on how often a domain is cited.
• Chrome treats it as optional. Lighthouse has an llms.txt audit in its agentic-browsing category, and a missing file is reported as “Not Applicable”.

That second-place fetcher is the real use case: coding agents pull llms.txt to read documentation. A related idea is worth more of your time: serving Markdown to agents that ask for it. Cloudflare's Markdown for Agents answers an Accept: text/markdown request with a Markdown version of the page.

Verdict: for a local business site, llms.txt is a five-minute nicety. We publish one on our own site because it costs nothing. Don't sell it as the thing that gets a business recommended.

Google is just as direct: “Structured data isn't required for generative AI search, and there's no special schema.org markup you need to add”. Its one rule for AI features is consistency: structured data should match the visible text on the page. Microsoft is warmer, calling schema “a type of code that helps search engines and AI systems understand your content”, while adding that nothing guarantees selection.

Two experiments are more useful than either statement:

• Ahrefs added JSON-LD to 1,885 pages and watched for citations. The changes were small and went both ways: AI Overviews −4.6%, AI Mode +2.4%, ChatGPT +2.2%. No clear effect.
• Mark Williams-Cook put a made-up address into invalid JSON-LD and nowhere else on the page. ChatGPT and Perplexity returned it. The model wasn't parsing schema; it was reading the HTML.

Put together, JSON-LD stops being magic. It's one more place where the facts sit as text: useful when they're right, harmful when they've drifted from the page. So ship the basics, and keep them true:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Dentist",
  "name": "Harbour Street Dental",
  "url": "https://example.com/",
  "telephone": "+44 1947 000000",
  "address": {
    "@type": "PostalAddress",
    "streetAddress": "12 Harbour Street",
    "addressLocality": "Whitby",
    "postalCode": "YO21 0AA",
    "addressCountry": "GB"
  },
  "openingHours": "Mo-Fr 09:00-17:30"
}
</script>

Use the most specific LocalBusiness subtype that fits, as Google advises. And don't count on FAQ markup for display: Google's FAQ rich results stopped appearing on 7 May 2026. Questions and answers as visible text are still worth having.

1. A firewall that blocks the crawlers you invited. Google's checklist asks that crawling is allowed “in robots.txt, and by any CDN or hosting infrastructure”, and since July 2025 Cloudflare has blocked AI crawlers by default, asking every new domain whether to let them in. OpenAI is clear about the stakes: sites that opt out of OAI-SearchBot “will not be shown in ChatGPT search answers”. A quick test:

for ua in OAI-SearchBot GPTBot ClaudeBot Claude-SearchBot PerplexityBot; do
  code=$(curl -s -o /dev/null -w '%{http_code}' \
    -A "Mozilla/5.0 (compatible; $ua/1.0)" https://example.com/)
  echo "$ua $code"
done

This only catches rules keyed on the user agent. A 200 doesn't prove the real crawler gets in, because some firewalls also check IP ranges. A 403 is a finding: confirm it in the server or CDN logs.

2. Pages that are empty without JavaScript. Vercel and MERJ found that none of the major AI crawlers rendered JavaScript, apart from Gemini, which uses Googlebot. Count the words in the raw HTML:

curl -s https://example.com/menu/ | sed 's/<[^>]*>/ /g' | wc -w

If the number is a few dozen and the page looks full in a browser, the content arrives later from an API, and most assistants never see it.

3. Facts that disagree. A phone number on the site that differs from the one on Google Maps. Two addresses. A Maps category that doesn't match what the business sells. OpenAI warns that “search results and citations can be incomplete, outdated, or incorrect”, and conflicting sources give it more ways to be wrong. The fix isn't a file. It's making the sources agree.

1. Search crawlers can fetch the site: robots.txt, firewall, logs.
2. Key facts are in the server-rendered HTML.
3. Facts match across the site, the Google Business Profile and the main directories.
4. JSON-LD repeats those same facts.
5. llms.txt and Markdown versions, if you have ten minutes left.

Steps 1 to 3 are where the problems usually are. Steps 4 and 5 are where people start. For more on step 2, see Server-render the facts.