llms.txt is widely recommended as a must-have 2026 AI search tactic, the AI-era equivalent of robots.txt. The data says otherwise. An Ahrefs analysis of 137,000 domains found 97% of llms.txt files received zero requests. This post walks through three independent studies that all point the same direction, explains the structural reason llms.txt does not behave like robots.txt, and covers what to implement instead for confirmed AI crawlability signals.
What llms.txt promises
An llms.txt file is a plain-text file placed at the root of a domain that lists the content an AI system is encouraged to prioritise. The format was proposed by Answer.AI in 2024, modelled on robots.txt: a file that tells AI systems what matters on a site, formatted for the way people assume language models read pages.
The comparison to robots.txt is where the confusion starts. robots.txt is a protocol with consequences: crawlers that ignore disallow rules risk being blocked at the network level, which creates a genuine compliance incentive. llms.txt is advisory text with no enforcement mechanism. Nothing requires an AI crawler to read it, let alone act on it.
Three independent studies, one direction
The strongest reason to be skeptical of llms.txt is not any single study. It is that three separate research teams, using three different methodologies, all found the same null result.
OtterlyAIran a 90-day experiment tracking 62,100+ total AI bot visits across sites with and without llms.txt files. Visits to /llms.txt itself totalled 84, just 0.1% of AI bot traffic, performing three times worse than the site's average content pages. The finding: no positive correlation between llms.txt presence and increased AI crawler activity.
SE Rankingtook a different approach: an XGBoost model trained to predict AI citation rates across approximately 300,000 domains. Removing llms.txt as a variable improved the model's prediction accuracy. If llms.txt carried real signal, removing it should have made the model worse, not better.
Ahrefs went straight to server logs: 137,000 domains, checking whether anything actually requested the file. 97% of llms.txt files got zero requests. Of roughly 38,000 domains with a valid file, only about 1,100 received any traffic at all, and of the requests that did land, 96% came from bots that were mostly not AI-related. Retrieval bots tied to ChatGPT and Perplexity, the ones that would actually generate a citation, made up just 1% of the fetches that happened.
| Study | Method | Finding |
|---|---|---|
| OtterlyAI | 90-day crawl tracking, 62,100+ AI bot visits | /llms.txt got 84 visits, 3x worse than average pages |
| SE Ranking | XGBoost citation model, ~300,000 domains | Removing llms.txt improved model accuracy |
| Ahrefs | Server log analysis, 137,000 domains | 97% of files received zero requests |
Source: OtterlyAI, SE Ranking, and Ahrefs, independent studies, 2025-2026.
Why llms.txt does not work the way robots.txt does
robots.txt has teeth. AI crawlers that ignore it risk being blocked outright, so there is a compliance incentive on the crawler's side. llms.txt has none: it is a suggestion with no protocol enforcement, and no major AI platform has published documentation confirming they parse it in the first place.
Google's John Mueller addressed this directly on r/TechSEO: "Google doesn't use llms.txt or llms-author.txt. I don't know of any other crawler / llm confirming they're using these (other than SEO tools)."
That is about as direct a disconfirmation as a platform statement gets. A file with no confirmed readers cannot produce a measurable citation effect, and the crawl data above is consistent with exactly that.
The one place llms.txt does show up in Google's tooling
This is where the picture gets more interesting than a flat "llms.txt is dead" take. Google Chrome's Lighthouse now ships an Agentic Browsing scoring category that checks for the presence of an llms.txt file, alongside signals like WebMCP integration and accessibility tree integrity. Google's own documentation notes that without llms.txt, "agents may spend more time crawling the site to understand its high-level structure."
(This is a different use case from AI search citation. Agentic Browsing scores how well a page supports a computer-use agent navigating it live, tools like Project Mariner, not whether an AI search engine selects your page as a source. It is a real, documented use for llms.txt. It is not the use case most sites adopt it for.)
What to implement instead
Redirect the effort into the three signals with confirmed evidence behind them.
1. Allowlist AI crawlers in robots.txt
GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, and Bingbot are each controlled independently and must be allowlisted individually. OtterlyAI found 73% of sites have at least one technical barrier blocking AI crawler access, and robots.txt misconfiguration is the largest single category. This is the confirmed access mechanism that llms.txt was mistaken for. See the full AI crawlability tactic for the complete allowlist.
2. Serve core content as static HTML
Five of seven major AI crawlers cannot render JavaScript. If your primary content is populated client-side, most AI crawlers never see it, regardless of what an llms.txt file claims about your site structure. Server-side rendering or static generation is the fix, and it is testable in under a minute: fetch a page with curl and confirm your key body text appears in the raw HTML.
3. Add schema markup where it is confirmed to correlate
SE Ranking found 81% of pages appearing in Google AI Overviews carry schema markup. FAQPage, Article, and HowTo are the types with the strongest association. Unlike llms.txt, this is a signal with a measured correlation to citation, not an unread text file. See the schema markup tactic for implementation details.
The bottom line
Three independent studies, using crawl tracking, predictive modelling, and raw server logs, all found the same thing: llms.txt has no measurable effect on AI crawler activity or citation rates. Google has stated directly that it does not use the file for search. The one confirmed use, Chrome's Agentic Browsing scoring, is a different problem than the one most sites adopt llms.txt to solve.
Adding an llms.txt file costs little, so this is not an urgent removal task if you already have one. But do not let it substitute for the three things the evidence actually supports: crawler allowlisting, static HTML, and schema markup. Those are where the citation gains are.