AI glossary · Web and SEO
What is llms.txt?
Also called: llms txt, LLMs.txt file
Definition
llms.txt is a proposed convention for a Markdown file at a website’s /llms.txt path that gives language models a short summary of the site and curated links to its most useful pages.
Explained
How it works
The proposal, published on llmstxt.org in September 2024, targets a practical problem: whole websites don’t fit in a context window, and HTML pages are full of navigation and scripts. A small Markdown index lets an assistant or coding agent find the right pages when it needs them. The proposal says its authors expected the file to be mainly useful at inference time, when a model answers, rather than for training, and that this is how it has been used.
The format is plain Markdown in a fixed order. An H1 with the site or project name is the only required part. Then come an optional blockquote summary, optional notes, and any number of H2 sections, each a list of [name](url): notes links. A section titled “Optional” marks links an agent can skip when it needs a shorter context. The proposal also suggests serving Markdown copies of pages at the same URL with .md appended, or with the extension replaced by .md.
It is not a standard, and it isn’t a replacement for robots.txt: it grants and blocks nothing.
Example
This site’s own llms.txt
This site publishes one at /llms.txt, generated when the site is built, from the same page registry as the sitemap, so it can’t list a page that doesn’t exist. It opens with the H1 and the summary, lists every live tool under its category with a one-line description, then every guide, error page, glossary term and directory, and ends with the about and policy pages. The top of the file looks like this, with ... marking lines left out here:
# AI Dev Toolkit
> Free AI developer tools: count tokens, estimate LLM API costs, compare models, and generate Claude Code and MCP configs. Private, with no sign-up.
...
## Tokens & Costs
- [AI Token Counter](https://tools.its-tahir.com/token-counter): Count tokens for GPT, Claude, Gemini, DeepSeek, Qwen and more.
...
## AI glossary
...Cost and quality
Why it matters
Google’s guide to its generative AI features says you don’t need llms.txt or similar files to appear in Google Search, and that creating them “will neither harm nor help” your visibility or rankings in Google Search. So treat it as a convenience for the tools that do fetch it, not as SEO.
If you publish one, generate it from your page data rather than writing it by hand, keep descriptions specific, and link to pages that answer questions on their own. A stale llms.txt that points at moved pages is worse than none.
Don’t mix up
Common confusions
- llms.txt vs robots.txt
- robots.txt, standardised in RFC 9309, tells crawlers which URLs they may fetch. llms.txt has no rules at all. To keep AI crawlers out, use robots.txt and, for bots that ignore it, a firewall rule.
- llms.txt vs sitemap.xml
- A sitemap lists every indexable URL in XML for search engines. llms.txt is a short, curated Markdown summary written for language models to read.
Go deeper
Try it and read more
- Free toolAI Crawler robots.txt GeneratorBlock or allow GPTBot, ClaudeBot and other AI crawlers.
- Guide · 11 min readHow to block AI crawlers with robots.txt (GPTBot, ClaudeBot and more)Block AI training crawlers like GPTBot and ClaudeBot but stay in AI search, with complete robots.txt examples, matching rules and what robots.txt can’t stop.
Related
Related terms
- AI crawlerAn AI crawler is an automated bot that fetches web pages for an AI company, whether to collect training data, to index pages for AI search answers, or to read a page that a user asked an assistant about.
- Context windowA context window is the maximum number of tokens a language model can work with in one request, counting the system prompt, tool definitions, conversation history, documents and the reply it writes.
- CLAUDE.mdCLAUDE.md is a Markdown file of project instructions that Claude Code loads into context at the start of every session, and AGENTS.md is the open, tool-neutral equivalent read by many coding agents.
- AI agentAn AI agent is a program in which a language model works towards a goal by repeatedly choosing a tool to call, reading the result and deciding the next step, until the task is done or a stop condition is reached.