Does llms.txt Actually Work? What 137K Sites Show

Every SEO newsletter spent 2025 telling you to add an llms.txt file to your site or get left behind in the AI search era. Ahrefs just checked whether anyone’s actually reading the file. Across 137,000 domains, in May 2026: 97% of the llms.txt files that sites published got zero requests. Not low traffic. Zero.

What the study actually measured

Ahrefs pulled server logs and live traffic data across 137,000 domains and found that 28% of them publish an llms.txt file, a real, measurable adoption number. Of that roughly 38,000, about 1,100 files received any request at all during May 2026. The rest sat there, unread, the entire month.

Here’s the part that should reset expectations further: of the requests that did land, the large majority came from bots, SEO audit tools, and scanners checking whether the file existed, not from AI systems using it to build an answer. Retrieval and citation crawlers, the ones actually deciding what gets quoted back to a user, accounted for roughly one in ninety of those requests. That’s not a rounding error. That’s most of the traffic being other people’s SEO tools checking your llms.txt, not AI systems reading it.

Google already told you to skip it

This isn’t just one study landing awkwardly. Google’s own public guidance says you don’t need to create special machine-readable AI text files, markup, or Markdown to show up in its generative AI search results, and llms.txt is specifically named in that list of things you can skip. John Mueller was more direct when someone pressed him on Reddit: if an AI platform that actually sends you clients asks for the file, build it. Otherwise, “prioritize needs before dreams.”

That’s a fair way to read the whole situation. Nobody’s claiming llms.txt actively hurts anything. The claim being walked back is the one that mattered: that publishing it does something measurable for AI search visibility. As of May 2026’s data, for most sites, it doesn’t.

Where it actually gets used

Here’s the finding that doesn’t fit the “llms.txt is dead” headline some outlets ran with. IDE coding agents fetch it routinely. Cursor, GitHub Copilot, Claude Code, Cline, and Windsurf all check for /llms.txt when a developer points them at a documentation site, because a clean markdown index of your docs is easier for an agent to work with than scraping rendered HTML, stripping navigation chrome, and guessing at what’s actually the content.

If you maintain a developer tool, an API, or documentation that people integrate against with AI coding assistants sitting next to them, that’s a real, present-day audience for the file. It’s just not the audience most of the hype was written for. The confusion comes from conflating “AI reads this” (true, for coding agents working against docs) with “AI search cites this” (not supported by the traffic data, as of now).

The format, if you’re building one

The spec, maintained at llmstxt.org, is intentionally minimal: a markdown file with an H1 naming your project, a blockquote summary, optional free-text context, and H2-delimited sections listing links with short descriptions. No XML, no required schema beyond that structure. If you’re building this for a documentation site specifically, that’s a reasonable investment of an afternoon, not a project.

Worth noting: the W3C opened a working draft in June 2026 aimed at formally standardizing the format, which suggests it’s not going away even as the near-term “does it move the needle” data comes back thin. Standards bodies don’t usually pick up conventions nobody’s using; they pick up conventions that are sticking around long enough to be worth reconciling into something more formal.

The same goes for SEO work more broadly: the things that move real numbers rarely look like the newest shiny convention, they look like the boring, well-measured basics done consistently. Our post on what 800 posts taught us about technical SEO measurement makes the longer version of that argument with a whole corpus instead of one file format.

The actual decision

Skip it if you’re chasing AI Overview citations or ChatGPT mentions specifically; that’s not what the current data supports, and Google is telling you the same thing directly. Build it if you run developer-facing documentation, because the coding agents your users already have open are checking for it. For most of the rest of the web, this is a low-cost, low-return afternoon project, not the urgent AI-SEO migration a lot of guides made it sound like. If you’re weighing where to actually spend GEO effort instead, answer engine optimization as a discipline is where the real, better-evidenced work is happening right now, and it’s worth reading before you spend the afternoon on this instead.