Websites are built for people. Navigation, scripts, cookie banners, and layout all exist to help a human find their way around. When an AI agent lands on the same page, most of that is noise it has to strip away before it can reach the one paragraph it actually needs. Every bit of that noise costs tokens, and it's one more chance for the agent to misread what you do.
llms.txt is a small proposal aimed at that problem: a Markdown file at the root of your site that hands large language models a curated map of your most important content. A basic version is quick to ship — and it's become one of the most debated files in technical SEO.
The debate comes from a gap between publishing and reading. Plenty of sites now serve one, but far fewer AI systems verifiably fetch it. Our take: it's useful infrastructure for some sites and a distraction for others, and the difference comes down to who actually reads yours. Below, we cover what the file is, how it differs from robots.txt and sitemap.xml, how to build one, and where it earns its keep. If you're already weighing how your site should show up for AI agents, we're happy to talk it through.
llms.txt is a proposed standard: a Markdown file, usually served at /llms.txt, that gives AI tools a short summary of a site and links to clean versions of its key pages.
It complements the crawler files you already have instead of replacing them, since robots.txt controls crawler access and llms.txt only offers guidance.
The format is simple: an H1 title, a blockquote summary, optional notes, and H2 sections of annotated links.
Coding agents reading technical docs are its clearest real-world users today, while Google says the file isn't needed to appear in its AI search features.
The most durable implementation is generated from your CMS and treated as low-cost infrastructure that supports AEO without standing in for it.
llms.txt is a plain text file, written in Markdown, that tells language models and agents what a website is about and where its most useful content lives. Jeremy Howard, co-founder of Answer.AI, published the proposal on September 3, 2024, with the spec hosted at llmstxt.org.
The reasoning is practical. HTML is hard for a model to convert back into clean text, and context windows, even as they've grown, still can't hold most websites whole. A short, curated index lets an agent read one file, decide which links matter for the question in front of it, and fetch only those.
The proposal pairs the file with a second idea. Pages an agent might need should also exist as clean Markdown at the same URL with .md added, so page.html also lives at page.html.md (or page.md).
In August 2026, Howard published v2 of the spec. It adds a discovery mechanism (covered below) and clarifies that a file at /docs/llms.txt covers everything under /docs/. It's still a community proposal maintained in the open on GitHub, with no formal standards body behind it.
All three files live at predictable paths in your root directory, which is why llms.txt often gets pitched as "robots.txt for AI." That shorthand is misleading because each file does a different job.
robots.txt is about access. It tells crawlers which paths they may request. It's also where you allow or block specific AI crawlers by user agent, like GPTBot for OpenAI (which runs separate agents for ChatGPT search and browsing), ClaudeBot for Anthropic (which also runs Claude-SearchBot for search indexing and Claude-User for fetches a user triggers), and PerplexityBot for Perplexity. Google-Extended works differently. It's a robots.txt token rather than a separate crawler, and it controls whether content Googlebot already crawls can be used to improve Google's Gemini models.
sitemap.xml is about coverage. XML sitemaps list every indexable page so search engines can find and refresh them. That's useful for SERPs, but a full sitemap is too large and unfiltered to drop into an agent's working context.
llms.txt is about guidance. It's a curated shortlist of what matters most, with a line of context on each link. Nothing in it grants or blocks access. If robots.txt disallows a bot, an llms.txt file won't change that.
A simple way to keep them straight: robots.txt says who can crawl what, sitemap.xml lists everything that exists, and llms.txt points to what an agent should read first. Most sites that publish llms.txt keep all three, and the spec was designed to sit alongside XML sitemaps rather than compete with them.
The format is deliberately minimal. Per the spec, sections appear in this order:
An H1 title. The name of the site or project, and the only required element.
A blockquote summary. One or two sentences with the context an agent needs to interpret everything else.
Optional notes. Paragraphs or lists (no headings) with extra guidance, like what a product does and doesn't support.
H2 sections of links. Each holds a Markdown list of \name\ links, optionally followed by a colon and a short description.
By convention, a section titled "Optional" holds secondary links an agent can skip when it's short on room. Here's a simplified example for a fictional analytics company:
\# Northwind Analytics
\> Northwind Analytics is a reporting platform for logistics teams. This file links to Markdown versions of our product docs, API reference, and company policies.
\> Northwind Analytics is a reporting platform for logistics teams. This file links to Markdown versions of our product docs, API reference, and company policies.
The REST API is versioned. v3 is current; v1 and v2 are deprecated.
\## Docs - Quickstart\: Connect a data source and build a first dashboard - API reference\: Endpoints, authentication, and rate limits for API v3
\## Company - Security and compliance\: Data handling and hosting regions - Integrations\: Supported data sources and setup notes
\## Optional - Changelog\: Release notes by version
Two details matter more than they look. Links should use real Markdown link syntax, because that's what parsers look for. And each description should say plainly what's on the page, since that one line is what an agent uses to decide whether to fetch it.
You'll often see a second file next to the index. If llms.txt is a table of contents, llms-full.txt is the whole book — the full text of a site's key pages concatenated into a single Markdown file.
Mintlify developed llms-full.txt with Anthropic, which wanted a cleaner way to feed its full documentation to models without parsing HTML. Because the platform generated it for every docs site it hosted, the pattern spread quickly. The current v2 spec text doesn't mention llms-full.txt, though, so treat it as a widely used convention rather than part of the proposal itself.
The trade-off is size. An llms-full.txt for a large documentation site can run to many megabytes, well past what fits in a single context window. For most marketing sites, the index alone is enough. For developer documentation, publishing both and linking llms-full.txt from llms.txt gives agents the choice.
The right setup depends on your stack and how often your content changes. We don't prescribe a specific tool. We look at how a site is built and maintained, then recommend an approach from there. Most implementations still come together through moves like these.
Pick a generation approach. Hand-writing works for a small site with stable pages. Docs platforms like Mintlify and GitBook generate the file automatically, and plugins exist for WordPress, Docusaurus, VitePress, and Drupal. An llms.txt generator that crawls your site can produce a first draft, but review it: automated output tends to list everything instead of curating.
Build it from your CMS when you can. The most durable version comes from the same content model that renders your pages, so titles, URLs, and descriptions update when the page does. A hand-maintained file drifts the first time someone renames a page. On a custom CMS, that's often a small build step: a template that queries published pages by section and outputs Markdown.
Serve clean Markdown versions. An index is only as useful as what it links to. Pointing agents at stripped-down .md versions of key pages is where much of the practical value lives, and many static site generators can output them at build time.
Publish at the root and add discovery links. Serve the file at /llms.txt as plain text. v2 also recommends standard link relations so agents can find it from any page: rel="describedby" points to the llms.txt that covers a page, and rel="alternate" type="text/markdown" points to that page's Markdown version. Both can go in an HTML <link> element or in an HTTP header set at the CDN:
Link: </docs/page.md>; rel="alternate"; type="text/markdown", </llms.txt>; rel="describedby"
Treat it like code. Agents are built to trust this file, and Ahrefs found researchers already probing llms.txt as a prompt-injection target. Keep it in version control, limit who can edit it, review anything a platform generates for you, and keep notes factual rather than instruction-shaped.
Test it. Chrome's Lighthouse now includes an llms.txt audit. It flags server errors when fetching the file and marks a missing file as not applicable. The spec suggests a more human check too: give an agent only your llms.txt and ask it questions about your business. Then watch your server logs to see what actually requests the file, and fold that into how you track AI search visibility overall.
This is the question most teams are really asking, and the honest answer depends on who you expect to read the file.
For AI search, the evidence is thin. Ahrefs studied 137,000 domains in its Web Analytics data and published the results in June 2026. About 28% published an llms.txt file (an upper bound, since those customers skew technical), and 97% of those files got zero requests in May 2026. AI search retrieval bots made up only about 1% of the requests that did arrive, and no AI bots went looking for the file on sites that didn't have one.
Google's position is direct. Its Search Central guide to generative AI features says site owners don't need llms.txt or other special files to appear in AI Overviews or AI Mode. Meanwhile, Chrome's Lighthouse team now audits for it as an agentic browsing signal — so even Google's guidance splits by product.
For agents doing real work, the picture is better. The v2 spec notes that coding agents use llms.txt files reliably, and among the AI tools Ahrefs did see, agents like Claude Code out-fetched AI search crawlers by a wide margin. That matches where the file started: developer documentation that coding assistants and MCP server integrations fetch on demand.
So the decision usually comes down to what your site is:
API products and docs sites: Worth doing well, with Markdown page versions and possibly llms-full.txt. This is where agents demonstrably use it.
SaaS and product sites with complex offerings: A curated file is cheap and can help browsing agents orient themselves, but don't expect it to move rankings.
Marketing and content sites: Fine to publish, especially if your CMS can generate it, but expect little measurable impact today. The fundamentals of answer engine optimization come first.
Not sure which bucket your site falls into? We can look at how it's built and talk through what's actually worth doing.
We don't treat llms.txt as an AEO lever. We treat it as one piece of a site that serves two audiences: the people who visit it and the agents that read it on their behalf. Visibility in AI search still rests on content that's clear, credible, and well structured, and what AI search actually changes has far more to do with that than with any single file.
What llms.txt does well is make your existing structure explicit. If your information architecture is a mess, writing the file will expose it fast, which is a useful exercise on its own. (We found that out rebuilding our own llms.txt from a sitemap dump into a curated map.) For teams publishing LLM-friendly content across docs, product pages, and a blog, the file becomes an honest index of what you want machines to read first. Keep expectations modest, generate it instead of hand-maintaining it, and revisit it as AI bots and agent tooling mature.
Strip away the hype, and llms.txt is a useful convention for one job: giving AI agents a clean, curated path into your content. For developer-facing products, that job matters now. For most other sites, it's low-cost groundwork that only pays off when the content and structure underneath it are strong.
That foundation is where we focus. For 15+ years, Big Human's strategists, designers, and engineers have built sites and products with content structure designed in from the start, from custom web development to AI web development and SEO content strategy. If you're working out how your site should show up in answer engines and agent workflows, let's talk about what you're building.