llms.txt · AI Crawlers · GEO

llms.txt in 2026: What the Server Logs Actually Show

Few files have generated as much debate in the SEO and GEO world over the past 18 months as llms.txt. Time to put the marketing slides aside and look at server logs — other people's, and our own.

Andreas Reuter · June 2026 · 6 min read

Few files have generated as much debate in the SEO and GEO world over the past 18 months as llms.txt. Some call it "robots.txt for AI"; others compare it to the long-buried keywords meta tag. Time to put the marketing slides aside and look at server logs — other people's, and our own.

What is llms.txt, anyway?

llms.txt is a Markdown file at the root of a domain (https://example.com/llms.txt). Its purpose is to give AI systems a curated overview of a website's most important content: links with one-line descriptions, optionally a short brand summary. The proposal comes from Jeremy Howard and was published in September 2024. Unlike robots.txt, llms.txt doesn't control access — it's an editorial offer: "Dear AI, this is what matters here."

The promise

The GEO marketing narrative goes like this: ship an llms.txt, and AI crawlers will understand you better, cite you more often, and send you more traffic from ChatGPT, Claude, Perplexity & co. Sounds plausible. But it can be measured.

The data in 2026

Adoption: roughly one in ten websites. An SE Ranking study of 300,000 domains in early 2026 found an adoption rate of about 10%. After 18 months of industry discussion, that's solid — but no breakthrough. Notably, large brands adopt at much higher rates than the average: right now, llms.txt is more of a sophistication signal than a standard.

Crawler interest: close to zero. The hardest numbers come from log analyses. One study covering more than 500 million AI bot events over 90 days found just 408 direct requests for llms.txt files. A second, independent measurement across three months and 60,000+ AI bot hits put it at 0.1% of visits. GPTBot, ClaudeBot, PerplexityBot, and Google-Extended overwhelmingly crawl regular HTML — not the curated shortcut.

Traffic impact: not demonstrable. Search Engine Land tracked ten websites for 90 days before and after implementation. Eight showed no measurable change. The two sites with traffic gains had other initiatives running in the same window (including a major PR campaign) — correlation, not causation.

The platforms are split. In July 2025, Google officially stated it does not support llms.txt — John Mueller even compared the file to the keywords meta tag. Anthropic and Perplexity, on the other hand, have publicly confirmed support, while OpenAI shows observable behavior but no commitment. Meanwhile, Yoast SEO now auto-generates llms.txt for WordPress sites — when a plugin with millions of installs ships a feature by default, mainstream adoption begins whether the crawlers care or not.

Our own logs

Theory is good; your own data is better. We analyzed the Nginx logs of robotcheck.coffee:

Not one of the 1,252 AI bot requests targeted /llms.txt. All 33 hits on that path came from monitoring tools and our own tests. Result: AI bots crawl regular HTML — the curated shortcut is consistently ignored.

This is a small website with modest traffic — not a 500-million-event study. But that's exactly why the result matters: it shows what reality looks like for a typical small-to-medium site, not for Stripe or Cloudflare.

What llms.txt is actually good for in 2026

The honest assessment: llms.txt is not (yet) an SEO or GEO lever. The file is a bet on the agentic web — a standardized, machine-readable surface AI agents may navigate on in the future. Where it's measurably useful today is developer documentation: AI-powered IDEs and coding agents genuinely use llms.txt to read docs efficiently.

The effort is minimal, and it does no harm. But if you expect a single Markdown file to transform your AI visibility, you're measuring the wrong end.

What works instead

AI crawlers consume what every crawler consumes: regular HTML. What matters is whether that layer is clean —

These are exactly the criteria our free check evaluates. Ship an llms.txt: sure, it takes half an hour. But only once the foundation is solid.

Check for free now →