Does adding an llms.txt file improve AI search visibility?

Key findings

What is an llms.txt file and what does it tell AI systems?

An llms.txt file is a plain-text file placed at the root of a domain (e.g., yourdomain.com/llms.txt) that lists the content an AI system is encouraged to prioritise. The format, proposed by Answer.AI, includes the domain's main purpose, key pages, and optionally a longer-form llms-full.txt with full documentation. The concept is modelled on robots.txt: which AI crawlers do respect: but with the important difference that robots.txt creates actual crawl constraints (bots follow it or risk being blocked), while llms.txt is advisory only and currently has no confirmed implementation by any major AI platform.

The distinction matters: robots.txt is a protocol with consequences for non-compliance. llms.txt is a convention with no enforcement mechanism. AI crawlers that encounter llms.txt have no requirement to change their behaviour based on it, and current evidence suggests they don't.

Does adding an llms.txt file improve AI search visibility?

No: the current evidence shows no measurable positive effect on AI search visibility.

An OtterlyAI study tracked 62,100+ AI bot visits over 90 days across sites with and without llms.txt files. The finding: no positive correlation between llms.txt presence and increased AI crawler activity or citation rates.

An SE Ranking XGBoost model trained to predict AI citation rates on approximately 300,000 domains found removing llms.txt as a variable actually improved prediction accuracy: suggesting the file provides no signal that helps predict whether a site gets cited. If llms.txt were working as intended, removing it from a predictive model would reduce accuracy, not improve it.

Why llms.txt doesn't work the way robots.txt does

robots.txt works because it has consequences: AI crawlers that ignore robots.txt risk being blocked at the network level, so there is compliance incentive. llms.txt is advisory. It tells AI systems what content to prioritise: but provides no mechanism to enforce that priority, and no major AI platform has published documentation confirming they parse or act on llms.txt content.

Answer.AI proposed the format in 2024. As of 2026, neither OpenAI, Anthropic, Google, nor Perplexity has confirmed llms.txt implementation. A file with no confirmed readers produces no measurable effect.

What the format actually looks like

If you still want to ship one — as low-cost hygiene rather than an expected ranking lever, since it costs almost nothing and might matter later if a platform starts reading it — this is the minimal Answer.AI-proposed structure, placed at `yourdomain.com/llms.txt`:

# Your Site Name

> One-sentence description of what this site is and who it's for.

Optional longer context paragraph about the site's purpose, scope, and
what makes its content worth prioritising.

## Docs

- [Getting Started](https://yourdomain.com/docs/getting-started): Short description of the page
- [API Reference](https://yourdomain.com/docs/api): Short description of the page

## Optional

- [Changelog](https://yourdomain.com/changelog): Lower-priority supplementary page

`betteraisearch.com/llms.txt` follows this same structure. Optionally pair it with a fuller `llms-full.txt` at the same path for longer-form documentation — same heading format, more sections. Neither file replaces robots.txt allowlisting or indexation in traditional search, which are the two confirmed access mechanisms covered in Implementation above.

The confusion with robots.txt

A Writesonic analysis of 6 AI crawlers found title tags (readable by 5/6 crawlers) and meta descriptions (readable by 4/6) are the actual high-readability content signals. JSON-LD structured data scored 0/6 for readability across all crawlers tested: and llms.txt, which is advisory plain text rather than structured HTML, is even further from how AI crawlers actually process pages.

AI crawlers read HTML. They follow HTTP responses. They respect robots.txt because robots.txt affects crawl access. They do not follow advisory text files with no protocol enforcement.

What to do instead

The fixes that actually improve AI crawler access: allowlist AI crawlers in robots.txt. PerplexityBot, GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, and Bingbot must each be explicitly allowlisted: each is controlled independently. Google officially added the Google-Agent crawler in 2026 for its AI systems; this must be allowlisted separately from GoogleBot.

73% of sites have at least one technical barrier blocking AI crawler access (OtterlyAI): robots.txt misconfiguration is the largest single category. Second, serve content as static HTML. Five of seven major AI crawlers cannot render JavaScript. If your content is generated client-side, AI systems may not be reading it regardless of what your llms.txt says.

What the evidence doesn't settle

The OtterlyAI and SE Ranking studies are negative results: they show llms.txt has no measurable effect based on current data. If major AI platforms implement llms.txt support in the future, the finding would need to be revisited.

The format may have value for internal documentation or communicating site structure to developers building AI integrations. That use case is separate from search citation visibility. For search citation purposes, the current evidence is clear: llms.txt is not a measurable AI search signal.

Does any research contradict this?

3 sources contradict this tactic. Consider these findings alongside the supporting evidence before acting.

Frequently asked questions

How this score is calculated

Each source is weighted by tier, independence, and how recent it is. Open the breakdown to see every source and how the score has changed over time.

Sources

Related analysis