llms.txt explained: what it is and how to create one

llms.txt is one of the most talked-about and least understood files on the web right now. The pitch is simple: drop a plain-text Markdown file at your domain root, and AI models like ChatGPT, Claude, and Perplexity will read a clean, curated summary of your site instead of fighting through cluttered HTML. The reality is more interesting than the pitch, because the file is advisory. Nothing forces an AI crawler to fetch it, and the 2025-2026 evidence on whether they actually do is mixed at best.

This guide covers what llms.txt is, how to write one in a few minutes, and the part most articles skip: a straight, evidence-based answer to whether it works. The short version is that llms.txt is cheap to publish and occasionally useful, but it is not a ranking button, and the only way to know if it helps is to measure whether AI crawlers request it.

What llms.txt is

llms.txt is a proposed standard for a Markdown file, served at yourdomain.com/llms.txt, that gives large language models a curated map of your most important content. It was proposed by Jeremy Howard of Answer.AI on September 3, 2024, and published at llmstxt.org. The goal is to help models “use a website at inference time” by handing them clean, linkable context rather than making them parse navigation menus, cookie banners, and ad markup.

The file solves a specific problem. LLMs work within a limited context window, and raw HTML burns tokens on markup that has nothing to do with your actual content. A well-structured llms.txt points a model straight to your documentation, product pages, or policies in a format it can ingest cheaply. Think of it as a reading list you hand to an AI assistant, curated by you rather than guessed at by a crawler.

It helps to place llms.txt next to the two files it is often compared with. Each does a different job:

File	Purpose	Who reads it
robots.txt	Grants or denies crawler access to paths	Search and AI crawlers
sitemap.xml	Lists every indexable URL for discovery	Search engines
llms.txt	Highlights and summarizes your best content	AI models (voluntarily)

The critical difference is enforcement. robots.txt is an instruction that well-behaved crawlers obey, and sitemap.xml is consumed by search engines that have committed to reading it. llms.txt is neither. It is a suggestion that a model can read, ignore, or never request in the first place. That single fact drives everything else in this article.

What goes in the file

An llms.txt file follows a strict but tiny Markdown structure, and only the first line is required. The spec at llmstxt.org defines the format in order:

• An H1 with the name of the project or site. This is the only mandatory element.
• A blockquote with a short summary of what the site is and who it serves.
• Zero or more Markdown sections with detail, guidance, or context.
• Zero or more H2 headings, each introducing a list of links in [name](url) format with optional notes.
• An optional section titled “Optional” for secondary links a model can skip when context is tight.

There is also a companion convention. llms-full.txt compiles your core content into one large Markdown file so a model can load everything at once, and some tools generate llms-ctx.txt variants for context windows. For most sites, a lean llms.txt with good links is the right starting point, not a full content dump.

How to create an llms.txt file

Creating an llms.txt file takes minutes: write the Markdown, save it as llms.txt, and upload it to your web root so it resolves at https://yourdomain.com/llms.txt. There is no registration, no schema validation service, and no submission step. Here is a complete, copy-paste example you can adapt:

# Lume

> Lume shows site owners the non-human half of their web traffic and
> cryptographically verifies which AI agents and crawlers are authentic.

Lume is a server-side analytics platform for AI-agent, bot, and human
traffic. It identifies crawlers by user agent, network, and signature,
then verifies whether each claimed agent is genuine.

## Docs

- [Getting started](https://example.com/docs/quickstart): Install the
  SDK and start capturing agent traffic in minutes.
- [Agent verification](https://example.com/docs/verify): How Lume
  confirms an agent is who it claims to be.
- [API reference](https://example.com/docs/api): Endpoints, auth, and
  response schemas.

## Guides

- [Reading your AI crawler report](https://example.com/guides/report)
- [Blocking vs. allowing agents](https://example.com/guides/policy)

## Optional

- [Changelog](https://example.com/changelog)
- [Brand assets](https://example.com/brand)

A few practical rules keep the file useful. Link only to canonical, high-value pages, because a bloated file defeats the purpose. Keep summaries short and factual, since models read them literally. Point to Markdown or clean HTML versions of pages where you have them, as several AI coding tools prefer Markdown responses. And treat the file as living documentation: if your docs move, update llms.txt in the same change, or you will point models at stale or 404 URLs. That last mistake is common enough that it has caused AI coding assistants to hallucinate broken links from outdated files.

Does llms.txt actually work?

This is the question that matters, and the honest answer as of mid-2026 is: mostly not yet, with narrow exceptions. The file is trivial to create, which is exactly why so many sites have one and so few see any benefit. The evidence falls into three buckets.

Major AI companies have not committed to it. OpenAI, Anthropic, and Google have not formally adopted llms.txt as a signal their production crawlers parse. Google has been the most vocal. Search Advocate John Mueller compared llms.txt to the long-dead keywords meta tag, noting that “none of the AI services have said they’re using llms.txt, and you can tell from server logs that they don’t even check for it.” Gary Illyes reinforced in 2025 that Google does not support the file and has no plans to. Google’s own Lighthouse documentation treats a missing llms.txt as “Not Applicable” and makes no claim that it affects Search or AI Overviews.

The server-log data is brutal. Ahrefs studied 137,000 domains and found that while 28% published a valid llms.txt file, 97% of those files received zero requests during May 2026. Separately, an analysis of more than 515 million LLM-bot traffic events reported that the crawlers that actually drive AI citations, including GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot, overwhelmingly skip llms.txt and crawl HTML directly. When Mueller says the bots do not check for the file, the logs largely agree.

There is a real, narrower use case. The counter-evidence comes from the documentation world. Mintlify, which hosts docs for Anthropic, Cloudflare, Vercel, and Cursor, points to data from answer-engine platforms showing AI agents visiting llms-full.txt more than twice as often as llms.txt, and to LangChain benchmarks where agents given an llms.txt outperformed vector search and context-stuffing on documentation questions. The pattern is consistent: llms.txt earns its keep for developer-facing docs that AI coding tools pull into context on demand, and it does close to nothing for a marketing site hoping to rank in AI answers.

Put those together and the verdict is clear. llms.txt is not a magic ranking button, and no major search or answer engine treats it as one. It is a low-cost, low-risk file that helps a specific audience (AI tools reasoning over your documentation) and is widely ignored by the crawlers that feed AI Overviews and chatbots. If you want to influence how you show up in generative results, the durable levers are original content and clean, crawlable pages, which is the core of generative engine optimization and the practical steps to optimize your site for AI search. llms.txt is a small complement to that work, not a substitute for it.

How to know if it is working on your site

The only reliable way to judge llms.txt is to measure whether AI crawlers actually request it, because industry averages tell you nothing about your specific domain. This is where server-side observation beats speculation. If your logs show GPTBot, ClaudeBot, or PerplexityBot fetching /llms.txt, the file is doing something for you. If they only ever pull HTML, you have your answer, and you can stop worrying about it.

This is exactly the gap Lume was built to close. Lume shows you the non-human half of your traffic server-side and identifies which AI crawlers hit your site and what they request, so you can see whether a given bot ever touches /llms.txt versus crawling pages directly. It also verifies whether an agent claiming to be GPTBot or ClaudeBot is authentic rather than a spoofed user agent, which matters because roughly a fifth of “AI crawler” hits are impostors. Pairing a published llms.txt with real agent analytics turns the question from “should I have this file” into “is this file being read,” which is the only version of the question worth answering. The same visibility lets you connect crawler behavior to outcomes when you track AI referral traffic arriving from those tools.

Should you create one anyway?

For most sites, yes, publish a lean llms.txt, because the downside is nearly zero and the file occasionally helps. It costs a few minutes, it will not hurt your rankings, and if you run documentation that AI coding assistants consume, it can measurably improve the answers those tools give. Just calibrate your expectations. Publish a tight file that links to your genuinely best pages, keep it current, and do not expect it to move you up in ChatGPT or Google AI Overviews on its own.

Skip the maximalist approach. You do not need a giant llms-full.txt dump, a file crammed with every URL, or a paid “AI visibility” service built around it. Publish the minimal version, then watch your logs. The moment you can see which agents fetch it, you replace a year of speculation with evidence from your own traffic, and you can decide whether to invest further based on what the crawlers on your domain actually do.

Frequently asked questions

What is an llms.txt file?

An llms.txt file is a Markdown document served at your domain root that gives AI models a curated summary and link list of your most important content. It is a proposed standard, published in September 2024, designed to help LLMs read your site efficiently within a limited context window.

Where does llms.txt go on my site?

It goes at the root of your domain, so it resolves at https://yourdomain.com/llms.txt. This mirrors the placement of robots.txt and sitemap.xml. Upload it to your web root or configure your framework to serve it at that path.

Does Google use llms.txt for rankings?

No. Google representatives John Mueller and Gary Illyes have both stated Google does not use llms.txt, with Mueller comparing it to the discredited keywords meta tag. Google’s Lighthouse documentation confirms the file has no effect on Search or AI Overviews.

Is llms.txt the same as robots.txt?

No. robots.txt controls whether crawlers may access paths and is obeyed by well-behaved bots. llms.txt is purely advisory and only summarizes your best content for models that choose to read it. Neither replaces the other, and llms.txt does not block or grant access to anything.

Do AI crawlers actually read llms.txt?

Rarely, based on current data. A 137,000-domain study found 97% of published files received zero requests in May 2026, and log analysis shows the major citation-driving crawlers usually skip the file and read HTML directly. The clearest exceptions are AI coding tools reasoning over developer documentation. The only way to know for your site is to check your own server logs or use agent analytics and AI-referral tracking to see which bots request it.

What should I put in llms.txt?

Start with an H1 site name, a one-line blockquote summary, and H2 sections linking to your best documentation, guides, or product pages in Markdown link format. Keep it short, link only to canonical pages, and use an “Optional” section for lower-priority links a model can skip.