The Complete Guide to llms.txt: Should Your Website Have One?

As brands race to optimize for AI visibility, a new file format has entered the conversation: llms.txt. Proposed by Jeremy Howard, co-founder of Answer.AI, in September 2024, llms.txt aims to do for AI language models what robots.txt did for search engine crawlers -- provide a standardized way for websites to communicate with machines.

The proposal has generated significant discussion in the SEO and AI communities. But the critical question remains: does it actually work? Here is what the data says.

What Is llms.txt?

The llms.txt file is a markdown-formatted file placed at the root of a website (for example, yoursite.com/llms.txt) that provides structured information to help large language models understand and use the site's content during inference.

Unlike robots.txt, which tells crawlers what to avoid, llms.txt tells AI systems what to prioritize. It is designed to be read by AI models at the point of generating a response, not during the crawling and indexing phase.

The specification defines a simple structure. A required H1 heading with the project or site name. An optional blockquote providing a concise summary. Optional body content with detailed information in markdown format. Optional file lists under H2 headings containing curated URLs with descriptions, formatted as markdown hyperlinks followed by a colon and explanatory notes.

The format also supports an "Optional" section under a dedicated H2 heading. URLs listed there indicate secondary content that can be skipped if the AI needs a shorter context window. Additionally, the specification recommends that websites provide clean markdown versions of pages by appending .md to existing URLs (for example, /about.html.md), making it easier for AI systems to process content without parsing complex HTML.

The deliberate choice of markdown over XML or JSON reflects the reality that language models are already highly proficient at processing markdown text.

Current Adoption Rates

Adoption of llms.txt remains limited. According to SE Ranking's analysis of 300,000 domains, the adoption rate stands at approximately 10.13%. This is notably low considering the intensity of interest in AI search optimization.

An interesting pattern has emerged in who is adopting it. Smaller and mid-sized websites show higher adoption rates than established authority sites with larger budgets and development teams. The most significant adoption event occurred in November 2024 when Mintlify rolled out llms.txt support across all documentation sites it hosts. This single action brought thousands of sites into compliance overnight, including documentation for Anthropic and Cursor.

Other notable implementations include Cloudflare's developer documentation and Hugging Face. However, these are predominantly developer-focused, documentation-heavy sites. Adoption among commercial websites, e-commerce, media publishers, and enterprise brands remains minimal.

Does It Actually Affect AI Citations?

This is where the picture gets sobering. As of early 2026, there is no confirmed evidence that implementing llms.txt directly improves AI citation rates.

No major LLM provider has confirmed support for the format. OpenAI makes no mention of llms.txt in its GPTBot crawler documentation. Google added llms.txt to some developer sites in December 2024, then removed the files within 24 hours. Meta has not acknowledged the format. Anthropic endorsed it in November 2024, but the degree to which Claude actively uses the file during inference remains unclear.

Search Engine Land conducted a study and reported that 8 out of 9 sites saw no measurable change in traffic after implementing llms.txt. Google's John Mueller stated publicly that, to his knowledge, none of the AI services have said they use llms.txt, and that server logs confirm most AI crawlers do not even request the file.

One counterpoint: Ray Martinez observed GPTBot crawling his llms.txt file immediately after implementation, and Mintlify reported 436 AI crawler visits to llms.txt files across their hosted sites. These data points suggest that some crawlers are at least aware of the file, even if they are not confirmed to use it for citation decisions.

The Case For Implementing llms.txt

Despite the lack of confirmed impact, there are reasonable arguments for implementation.

The cost is minimal. Creating a well-structured llms.txt file takes less than an hour for most websites. The maintenance burden is negligible. If the format does gain traction with AI providers, early adopters will already be in compliance.

The file serves as good documentation practice regardless of AI impact. Forcing yourself to create a concise, structured summary of your site's most important content and resources is a useful exercise that can inform other optimization work.

There is also a signaling effect. Having llms.txt demonstrates to AI-savvy visitors and potential partners that your organization is forward-thinking about AI accessibility.

The Case Against Prioritizing llms.txt

The strongest argument against prioritizing llms.txt is opportunity cost. Development time spent on llms.txt is time not spent on optimization strategies with proven impact.

Structured data (JSON-LD schema markup), E-E-A-T signals, and GEO content optimization have well-documented, measurable effects on AI citation rates. The Princeton GEO research demonstrated up to 40% improvement in AI visibility from content optimization methods. Pages with well-organized headings are 2.8x more likely to earn citations. Content with statistics and citations achieves 30-40% higher visibility in AI responses.

These are proven levers. llms.txt is, at this point, speculative.

How to Create llms.txt for Your Site

If you decide to implement it, here is a practical template.

# Your Company Name

> Brief one-paragraph description of your company, what you do,
> and what makes you an authority in your space.

Key information about your organization, products, or services
that you want AI systems to understand when generating responses
about your industry.

## Documentation

- [Product Overview](/docs/overview): Complete guide to your product
- [API Reference](/docs/api): Technical API documentation
- [Case Studies](/case-studies): Real-world implementation examples
- [FAQ](/faq): Frequently asked questions and answers

## Optional

- [Blog](/blog): Industry insights and company news
- [Team](/about/team): Leadership and team information
- [Press](/press): Media coverage and press releases

Place this file at your site root so it is accessible at yoursite.com/llms.txt. Keep it concise, link to your most authoritative pages, and update it when your site structure changes significantly.

The Bottom Line

llms.txt is an interesting proposal that addresses a real need: helping AI systems understand website content more effectively. However, the data does not support treating it as a high-priority optimization.

If you have spare development time, implementing it is a low-risk, low-cost action. But it should not come at the expense of proven AI citation strategies: schema markup, E-E-A-T signal strengthening, GEO content optimization, and technical excellence.

The factors that actually drive AI citation today are well-established. CiteOps.ai analyzes all of them, including your robots.txt configuration for AI bot access, giving you a clear picture of what to prioritize. Focus on what works, and keep llms.txt on your radar as the standard matures. To understand the broader strategic picture, read our guide on GEO vs SEO and the step-by-step playbook for getting cited by ChatGPT.