In late 2024 a new proposal quietly appeared in the world of technical SEO: a plain text file called llms.txt. By 2026 it has become one of the most discussed standards in AI visibility, adopted by thousands of documentation platforms and checked by modern audit tools. This guide explains what llms.txt is, how it differs from robots.txt and sitemap.xml, and how to create one that actually helps AI systems understand your website.

What Is llms.txt?

The llms.txt file is a Markdown document placed at the root of your domain, for example yoursite.com/llms.txt. It was proposed in September 2024 by Jeremy Howard, co-founder of Answer.AI, as a way to give large language models a curated, machine friendly map of the most important content on a website.

The core problem it solves is context. When an AI assistant fetches a normal web page, it receives a large amount of HTML: navigation menus, cookie banners, scripts, footers and tracking code. The actual content is buried inside all of that markup. Language models also work with limited context windows, which means they cannot load an entire website at once. The llms.txt file gives them a short, clean starting point written in a format they parse extremely well: Markdown.

llms.txt vs robots.txt vs sitemap.xml

These three files are often confused, but they serve completely different purposes.

robots.txt is about permissions. It tells crawlers which parts of your site they may or may not access. It does not describe your content in any way. If you want the full picture, read our complete robots.txt guide.

sitemap.xml is an inventory. It lists every URL you want indexed, with no explanation of what each page contains or which pages matter most. It is designed for search engine crawlers with massive crawl budgets.

llms.txt is a guide. It is a short, human curated document that says: here is what this website is about, and here are the pages that matter most, each with a one line description. It is designed for systems that need to understand your site quickly, with as few requests as possible.

The Format Explained

The specification is intentionally simple. A valid llms.txt file contains an H1 title, an optional blockquote with a short summary, and one or more sections with lists of links. Here is a minimal example:

# Example Store

> Example Store sells handmade leather goods and ships worldwide. Founded in 2015, based in Vienna.

## Products
- [Wallets](https://example.com/wallets): Full grain leather wallets in 12 designs
- [Bags](https://example.com/bags): Messenger bags and backpacks

## Support
- [Shipping and returns](https://example.com/shipping): Delivery times and return policy
- [Contact](https://example.com/contact): Email and phone support details

## Optional
- [Company history](https://example.com/about): The story behind the brand

Two conventions are worth knowing. First, a section titled Optional marks links that can safely be skipped when an AI system is short on context space. Second, some sites also publish a companion file called llms-full.txt, which contains the complete text of key pages in a single document, so a model can read everything without crawling page by page.

Who Actually Supports It in 2026?

Adoption grew quickly through 2025 and 2026, especially among developer documentation platforms. Documentation hosts began generating the file automatically for every project, and major AI companies published llms.txt files for their own docs. Thousands of SaaS products, open source projects and technical blogs now serve one.

It is important to be honest about the other side. Google has publicly stated that its search systems do not currently use llms.txt, and some search engineers have compared it to the old keywords meta tag. OpenAI has not made any formal commitment either. What we observe in practice is different from formal commitments though: server logs across many sites show regular fetches of /llms.txt from AI related user agents, particularly the user triggered fetchers that retrieve pages live when someone asks an assistant a question. Those live retrieval systems benefit the most from a compact, curated summary.

Does It Actually Help?

Think of llms.txt as low cost insurance. Creating the file takes fifteen to thirty minutes, adds a few kilobytes to your server, and carries no penalty from any search engine. In exchange, any AI system that chooses to read it gets an accurate, up to date description of your business written by you, instead of guessing from scraped HTML fragments.

For documentation heavy sites, online stores with clear category structures, and service businesses with a handful of key pages, the file is genuinely useful. For a five page brochure site the benefit is smaller, but the cost is close to zero either way.

How to Create Your llms.txt Step by Step

Step 1: List the ten to thirty most valuable URLs on your site. Think about the pages you would show a new customer first.

Step 2: Write a one or two sentence factual summary of what your website or business does. Avoid marketing slogans, this text is for machines that reward clarity.

Step 3: Group the links into two to five logical sections, such as Products, Guides, Pricing and Support.

Step 4: Write a single factual description line for every link. Imagine the reader has zero prior context about your company.

Step 5: Save the file as plain UTF-8 text and upload it to the root of your domain. The server should return it with a text based content type, ideally text/plain or text/markdown.

Step 6: Test it. Open yoursite.com/llms.txt in a browser and view the source. You should see raw Markdown, not an HTML page.

Step 7: Put a reminder in your calendar to review the file every quarter, or whenever your site structure changes.

Common Mistakes to Avoid

Serving HTML instead of Markdown. This is by far the most common failure. Many content management systems catch unknown URLs and return a styled 404 page with a 200 status code. To a validator, that looks like an llms.txt file full of HTML soup. Always verify the raw response.

Dumping your entire sitemap into the file. A list of eight hundred URLs with no descriptions defeats the whole purpose. Curate ruthlessly.

Writing advertising copy. Descriptions like the best solution on the market tell a language model nothing. State plainly what each page contains.

Using relative URLs. Links in llms.txt should be absolute, including the https prefix, because the file may be read completely outside the context of your website.

Confusing it with ai.txt. An earlier proposal with a similar name existed but never gained adoption. We covered that story in our article about the ai.txt file. In 2026, llms.txt is the standard that actually gets fetched.

How to Check Yours in 30 Seconds

The CheckMy.site scanner tests for llms.txt as part of its AI Discovery category. It verifies that the file exists, that the response is real text rather than an HTML soft 404, and that the structure follows the expected Markdown format. The same scan also covers robots.txt rules for 18 AI crawlers, structured data, meta tags and over 150 other checks, so you see your complete AI visibility picture in one report.

The llms.txt standard is young, and nobody can promise it will become universal. But it is cheap, harmless and already read by real systems today. That combination makes it one of the easiest AI visibility wins available right now.