What is llms.txt
llms.txt is a proposed convention — think robots.txt, but for language models. You place a file at yoursite.com/llms.txt that contains a simplified, markdown-formatted summary of your site: what it is, what it does, links to key pages, and optionally a stripped-down version of your content that is easy for an LLM to parse without fighting through navigation, ads, cookie banners, and layout HTML.
The format was proposed by Jeremy Howard (Answer.AI / fast.ai) in September 2024, with a public specification at llmstxt.org. It is not an official standard. No standards body has ratified it. Most major LLMs do not specifically look for it. But a small and growing number of developer tools, documentation systems, and AI agents do.
What Google actually said
Google has been clear: llms.txt does not affect your search rankings. It is not a ranking signal. Googlebot does not treat it as a preferential crawl target. If you publish one, it will not help you rank higher on Google Search. At Search Central Live in July 2025, Google’s Gary Illyes stated that Google doesn’t support llms.txt and isn’t planning to, and Google’s official AI optimization guide says plainly that “you don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in generative AI search.”
This is accurate and important to say plainly, because a wave of SEO vendors has started selling llms.txt creation as an AI SEO service. It is not. For traditional search — and as we have established, traditional search is what feeds AI search results — llms.txt is irrelevant.
What a different part of Google is doing
Here is where it gets more nuanced. While Google Search dismisses llms.txt, Google’s Chrome team has quietly started checking for it. They added an “Agentic Browsing” audit category to Lighthouse — Google’s tooling for evaluating how well a site is built for machine interaction — and one of its deterministic checks is the presence of an llms.txt file. The stated rationale: without one, agents “may spend more time crawling the site to understand its high-level structure and primary content.”
So Google’s position is not contradictory. It is two separate statements about two separate use cases:
- For search rankings: llms.txt does not matter.
- For AI agents navigating your site: llms.txt is treated as useful reference material.
The confusion comes from treating these as the same claim.
Where llms.txt genuinely helps
The use case where llms.txt has real value is not SEO at all. It is documentation, developer tools, and technical reference sites — places where AI agents and coding assistants are the primary visitors, not human search users.
If you maintain an API, a developer SDK, or a technical documentation site, an AI agent trying to help a developer use your product will benefit significantly from a well-structured llms.txt. Instead of crawling hundreds of pages and trying to reconstruct your API’s structure, it reads your llms.txt and gets a clean map of what exists and where.
This is already how tools like Claude Code, Cursor, and GitHub Copilot access documentation when assisting with code. A developer asks their AI assistant how to authenticate with your API. The assistant fetches your docs. If your docs are structured clearly — and especially if llms.txt points it to the right pages — it gets a useful answer faster. If your docs are a maze of nested HTML with version selectors and pop-up login prompts, it may fail entirely.
For a B2B SaaS company with a developer audience, this is worth caring about. For a marketing site, it is largely irrelevant right now.
No major AI platform formally consumes it yet
It is worth being precise about where things stand. Google’s only engagement with llms.txt is in its agent-readiness tooling, not its search or AI answer products — Search and AI Overviews explicitly ignore it. And the other major AI players have not formally adopted the convention either. OpenAI has not published guidance on it. Anthropic has not. Perplexity has not. (Anthropic, Cursor, and others do publish an llms.txt for their own docs — but publishing one is not the same as the models systematically reading one.)
That limits its current reach. If you are optimising for visibility in ChatGPT or Claude, llms.txt is not the lever to pull. Your search ranking still determines that, regardless of any files in your root directory.
Why it might matter more later
This is my own read, not established fact: I think llms.txt becomes more valuable as AI harnesses become more prevalent.
A harness — Claude Code, Cursor, a custom AI agent built on the Anthropic API — can access URLs directly. When a developer uses one of these tools to research, build, or debug, the AI routinely fetches pages to read them. It is not using a search index. It is not Googlebot. It is an agent pulling a URL and reading whatever it finds.
As more work gets done through these harnesses, the way content gets consumed changes — and the volume is climbing fast. Cloudflare, which sees a large share of global web traffic, reported that OpenAI’s GPTBot grew its request volume 305% in a single year, moving from the ninth most active crawler to the third. An agent that fetches your documentation URL and hits a clean llms.txt gets oriented immediately. One that hits a wall of JavaScript-rendered HTML with no clear entry point gets less useful information and may hallucinate the rest.
If adoption grows — if more AI tools start looking for llms.txt as a convention the way browsers look for favicon.ico — then having one becomes an ambient advantage. Not a ranking signal. Not an SEO tactic. Just good practice for a web where agents are increasingly common visitors.
We are not there yet. But the direction of travel seems clear enough to make it worth implementing on documentation-heavy sites, and worth watching on everything else.
B2B SEO practitioner specialising in search strategy for the AI era. Working directly with marketing managers at mid-size companies — no account managers, no handoffs.