llms.txt is widely recommended as a must-have 2026 AI search tactic, the AI-era equivalent of robots.txt. The data says otherwise. An Ahrefs analysis of 137,000 domains found only about 38,000 had a valid llms.txt file at all, and 97% of those received zero requests. This post walks through three independent studies that all point the same direction, explains the structural reason llms.txt does not behave like robots.txt, and covers what to implement instead for confirmed AI crawlability signals.
What is llms.txt supposed to do?
An llms.txt file is a plain-text file placed at the root of a domain that lists the content an AI system is encouraged to prioritise. The format was proposed by Answer.AI in 2024, modelled on robots.txt: a file that tells AI systems what matters on a site, formatted for the way people assume language models read pages.
The comparison to robots.txt is where the confusion starts: llms.txt is advisory text with no enforcement mechanism, while robots.txt carries a real compliance incentive on the crawler's side.
Ahrefs, OtterlyAI, and SE Ranking independently found the same null result
Three separate research teams (Ahrefs, OtterlyAI, and SE Ranking) used three different methodologies and reached the same null result. Methodological convergence like that is stronger evidence than any single study provides alone.
OtterlyAI ran a 90-day experiment tracking 62,100+ total AI bot visits across sites with and without llms.txt files. Visits to /llms.txt itself totalled 84, just 0.1% of AI bot traffic, performing three times worse than the site's average content pages.
OtterlyAI's finding: no positive correlation between llms.txt presence and increased AI crawler activity.
SE Ranking took a different approach: an XGBoost model trained to predict AI citation rates across approximately 300,000 domains. Removing llms.txt as a variable improved the model's prediction accuracy.
If llms.txt carried real signal, removing it should have made the model worse, not better.
Ahrefs went straight to server logs: 137,000 domains, checking whether anything actually requested the file. 97% of llms.txt files got zero requests. Of roughly 38,000 domains with a valid file, only about 1,100 received any traffic at all, and of the requests that did land, 96% came from bots that were mostly not AI-related.
Retrieval bots tied to ChatGPT and Perplexity, the ones that would actually generate a citation, made up just 1% of the fetches that happened.
| Study | Method | Finding |
|---|---|---|
| OtterlyAI | 90-day crawl tracking, 62,100+ AI bot visits | /llms.txt got 84 visits, 3x worse than average pages |
| SE Ranking | XGBoost citation model, ~300,000 domains | Removing llms.txt improved model accuracy |
| Ahrefs | Server log analysis, 137,000 domains | 97% of files received zero requests |
Source: OtterlyAI, SE Ranking, and Ahrefs, independent studies, 2025-2026.
Why does llms.txt not work the way robots.txt does?
llms.txt has no enforcement mechanism: no major AI platform has published documentation confirming they parse the file, let alone that they penalise sites that ignore it.
robots.txt is different: crawlers that ignore its disallow rules risk being blocked at the network level, which is why compliance with it is in the crawler's own interest.
Google's John Mueller addressed this directly on r/TechSEO: "Google doesn't use llms.txt or llms-author.txt. I don't know of any other crawler / llm confirming they're using these (other than SEO tools)."
Mueller's statement is about as direct a disconfirmation as a platform statement gets. A file with no confirmed readers cannot produce a measurable citation effect, and the crawl data above is consistent with exactly that.
Does Google use llms.txt for anything?
Chrome's Lighthouse tooling is where Google does use llms.txt: its newer Agentic Browsing scoring category checks for the presence of an llms.txt file, alongside signals like WebMCP integration and accessibility tree integrity. Google's own documentation notes that without llms.txt, "agents may spend more time crawling the site to understand its high-level structure."
(Agentic Browsing is a different use case from AI search citation: it scores how well a page supports a computer-use agent navigating it live, tools like Project Mariner, not whether an AI search engine selects your page as a source. It is a real, documented use for llms.txt, just not the one most sites adopt it for.)
What should you implement instead of llms.txt?
Crawler access comes first, static rendering second, and schema markup third, since an unreachable page is never crawled regardless of its markup. Fixing robots.txt allowlisting and JavaScript rendering has to happen before schema, which only correlates with citation once a page is actually crawlable. If you are choosing where to spend limited engineering time, fix the earliest blocker first, since nothing further down the list compounds until it is resolved.
1. Allowlist AI crawlers in robots.txt
Allowlisting GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, and Bingbot in robots.txt is the confirmed access mechanism that llms.txt was mistaken for, since each crawler is controlled independently and must be allowlisted separately. OtterlyAI found 73% of sites have at least one technical barrier blocking AI crawler access, with robots.txt misconfiguration the largest single category. See the full AI crawlability tactic for the complete allowlist.
2. Serve core content as static HTML
Five of seven major AI crawlers cannot render JavaScript. If your primary content is populated client-side, most AI crawlers never see it, regardless of what your llms.txt claims about site structure. Server-side rendering or static generation is the fix, and it is testable in under a minute: fetch a page with curl and confirm your key body text appears in the raw HTML.
3. Add schema markup where it is confirmed to correlate
SE Ranking found 81% of pages appearing in Google AI Overviews carry schema markup. FAQPage, Article, and HowTo are the types with the strongest association. Unlike llms.txt, this is a signal with a measured correlation to citation, not an unread text file. See the schema markup tactic for implementation details.
Three independent studies point the same direction: llms.txt shows no confirmed citation benefit
Three separate methodologies, crawl tracking, predictive modelling, and raw server logs, reached the same conclusion. None found llms.txt producing a measurable effect on AI crawler activity or citation rates.
Google has stated directly that it does not use the file for search. Chrome's Agentic Browsing scoring is a real, confirmed use for the file, but it solves a browsing-agent problem, not the AI-citation problem most sites install llms.txt to fix.
Adding llms.txt costs little, so this is not an urgent removal task if you already have one.
But do not let it substitute for the three things the evidence actually supports: crawler allowlisting, static HTML, and schema markup. Those are where the citation gains are.