What is llms.txt?
llms.txt is a machine-readable Markdown file designed to help language models and software agents find useful information on a website without first interpreting every part of the normal HTML interface.
The proposal was originally published by Jeremy Howard in September 2024. Version 2 was published in August 2026 after wider adoption and practical use. A site-wide file normally lives at /llms.txt, although version 2 also defines how an llms.txt file can sit inside a subpath and describe the material beneath that path.
The basic idea is curation. A sitemap might contain hundreds or thousands of URLs. An llms.txt file can instead identify the pages an agent is most likely to need, accompanied by concise context about what those pages contain.
The current llms.txt format
The specification uses Markdown rather than XML or JSON. That makes the file easy for a person to inspect while remaining straightforward for software to parse.
The only required part is an H1 containing the name of the project or website. The specification can then include a blockquote summarising the site, additional explanatory text and H2 sections containing lists of useful links.
A simple file might begin with a heading containing the company name, followed by a short sentence describing what the company does. A section called Services could then link to the principal service pages, while another section could point to technical documentation or policies.
The important part is selection. Copying an entire sitemap into llms.txt defeats much of its purpose. Useful entries should lead an agent towards authoritative pages that contain clear, maintained information.
llms.txt vs robots.txt and sitemap.xml
These files solve different problems and should not be used interchangeably.
- robots.txt communicates crawling instructions to automated user agents that choose to follow the Robots Exclusion Protocol.
- sitemap.xml helps search engines discover URLs and understand information such as update relationships supported by the sitemap format.
- llms.txt is a proposed convention for presenting selected site information and useful links in an LLM-friendly Markdown structure.
An llms.txt file does not grant crawler permission. If a bot is blocked in robots.txt or by a firewall, adding that bot's preferred pages to llms.txt does not override the block.
Likewise, robots.txt does not explain what a business does. It controls access. Businesses deciding which AI systems can crawl their content should review AI crawler management separately.
How to create a useful llms.txt file
Start with the information an external agent actually needs. For a service business, that normally means the organisation's identity, primary services, important factual pages and documentation that answers predictable questions.
- Publish the file at /llms.txt if it is intended to describe the whole site.
- Add the site or organisation name as the H1.
- Write a concise summary that states what the organisation does without promotional filler.
- Group important resources beneath descriptive H2 headings.
- Link to canonical, maintained pages rather than duplicate or temporary URLs.
- Fetch the public URL after publishing and confirm it returns the expected file with a successful HTTP response.
Version 2 also introduces standard link relationships that can help agents discover Markdown alternatives and the relevant llms.txt file. The specification describes rel="alternate" with type="text/markdown" for Markdown page versions and rel="describedby" for the llms.txt resource covering a page.
Those additions are useful on documentation-heavy sites, but they are not necessary merely to have a basic llms.txt file.
Does llms.txt improve AI search rankings?
There is no sound basis for promising that it does. Google explicitly advises site owners that unnecessary AI text files such as llms.txt are not required for its generative search features. For Google AI Overviews and AI Mode, the established requirements for normal Search remain the foundation.
That does not make llms.txt pointless. It can be a useful convention for agents that choose to look for it, particularly where a site has technical documentation or a large information architecture. The llms.txt project reports growing adoption and version 2 reflects lessons from sites and tools already using the convention.
The correct expectation is therefore modest: llms.txt can make selected information easier for compatible agents to locate and interpret. It cannot make a weak page authoritative, force an assistant to cite the site or replace conventional technical SEO.
Where llms.txt fits in an AI search strategy
Before spending time on llms.txt, check the fundamentals. Important pages should be publicly reachable, internally linked, technically indexable where appropriate and explicit about the facts users are trying to find.
A useful AI-search review also examines structured data, crawler controls, canonical URLs, content consistency and whether machines can access the same substantive information a human visitor sees. Those factors have a broader effect than a single supplementary text file.
We include llms.txt creation in our £499 AI Search Visibility Package, alongside the wider technical and content work. You can also create the file yourself using the specification linked below. There is no licence fee or platform fee for publishing your own llms.txt file.