LLMs.txt: Preparing Your Website for AI Search Engines

Let's start with the claim you actually came here to check: llms.txt is a proposal, not a standard, and as of mid-2026 no major AI company has publicly confirmed that its crawlers read it. Not OpenAI, not Anthropic, not Google, not Perplexity. If a guide tells you adding llms.txt will get you cited in ChatGPT, it is selling you certainty that doesn't exist yet. This post explains what the file is, what it realistically does and doesn't do, and where to spend your effort instead.

What llms.txt actually is

llms.txt is a convention proposed in 2024 by Jeremy Howard (of fast.ai). The idea: place a Markdown file at yourdomain.com/llms.txt that gives a large language model a clean, curated map of your site: a short description plus links to your most important pages, ideally in plain Markdown that's cheap to parse and fits in a context window.

A minimal file looks like this:

# Acme Analytics

> Self-serve product analytics for B2B SaaS teams.

## Docs
- [Quickstart](https://acme.com/docs/quickstart): set up in 10 minutes
- [API reference](https://acme.com/docs/api): full endpoint list

## Guides
- [Event tracking](https://acme.com/guides/events): naming conventions and schema

The intent is reasonable. LLMs work from a limited context window, and most web pages are wrapped in navigation, ads, and scripts that waste tokens. A curated Markdown index is, in principle, easier for a model to ingest than crawling your rendered HTML.

The honest status check

Here is where most articles overpromise. The reality in mid-2026:

• It is not an official standard. There is no W3C or IETF specification, no governing body, and no requirement that anyone honor it.
• Adoption by AI crawlers is unconfirmed. The companies that run the crawlers feeding ChatGPT, Claude, Gemini, and Perplexity have not announced that they fetch or use llms.txt. Absence of an announcement isn't proof they ignore it, but you should treat any "it works" claim without a primary source as marketing.
• It does nothing for Google Search rankings. This is not a Google-recognized file like robots.txt or sitemap.xml. It will not affect your blue-link rankings or your appearance in AI Overviews.
• Some developer-tool and docs sites have adopted it as a courtesy for users who paste docs into an AI assistant. That's a real use case, but it's "help humans feed your docs to a chatbot," not "rank in AI search."

None of this means the file is useless. It means the honest pitch is modest: it's low-effort, harmless, and might help, rather than a lever that will get you cited.

What genuinely helps an AI cite you

The mechanisms that actually get a page surfaced and quoted by answer engines today are the same ones that have always made content easy to extract, and they don't depend on any new file:

• A direct answer near the top. Models lift self-contained statements. A sentence like "A 301 redirect is permanent; a 302 is temporary" is quotable; three paragraphs of throat-clearing before the answer are not.
• Clean, parseable structure. Real headings, short paragraphs, tables, and lists let an extractor find the relevant chunk. This is the same work covered in the heading structure best practices post.
• An FAQ section. Question-and-answer pairs map almost perfectly onto the queries people type into AI tools, which is why they're cited disproportionately.
• Structured data. JSON-LD makes the meaning of your content machine-readable. It's worth doing for normal search regardless (see the schema markup guide), and a clean machine representation can only help an extractor.
• Being crawlable. If you block the AI crawlers in robots.txt (GPTBot, ClaudeBot, PerplexityBot, Google-Extended), no llms.txt will save you. Decide deliberately whether you want to be in AI answers, and set robots.txt to match; see the robots.txt guide.
• Factual accuracy and clear sourcing. Models prefer content that agrees with their other sources. Vague, hedged, or contradictory pages get skipped.

Notice that everything on this list is ordinary content quality. That's the real takeaway: the best "AI SEO" in 2026 is good SEO and good writing, structured so a machine can lift a clean answer.

So should you publish one?

Yes, with the right expectations. It costs ten minutes, can't hurt, and positions you if adoption grows. Keep it short and link only to your genuinely important pages; a bloated llms.txt defeats the entire point of being a clean, low-token map. You can build a starter file from your sitemap with our llms.txt generator, then trim it to the pages you'd actually want an AI to read first.

What you should not do is publish llms.txt and call your AI strategy done, or expect it to move search rankings, or pay anyone who guarantees citations because of it.

Frequently Asked Questions

Is llms.txt an official standard?

No. It's a community proposal from 2024 with no governing body, no formal specification, and no mandate. Treat it as an optional convention, not a requirement.

Do ChatGPT, Claude, Gemini, or Perplexity read llms.txt?

None of their operators have publicly confirmed that their crawlers fetch or use it. Anyone claiming definitively that a specific AI "reads your llms.txt" is going beyond what those companies have stated. It may help; it is not confirmed to.

Does llms.txt affect my Google rankings?

No. It's not a file Google Search recognizes, and it has no effect on your standard rankings or on whether you appear in AI Overviews. Those are driven by normal SEO and content quality.

Is llms.txt the same as robots.txt?

No, and they do opposite jobs. robots.txt is a recognized standard that controls which crawlers may access your pages. llms.txt is an unofficial proposal that tries to describe and index your content for models. robots.txt restricts; llms.txt advertises.

What actually gets my content cited by AI tools?

Clear, self-contained answers near the top of the page, real heading structure, FAQ sections, accurate and well-sourced facts, and not blocking AI crawlers in robots.txt. These are content-quality fundamentals, not a single file.

Should I block AI crawlers or invite them?

That's a business decision. Inviting them (the default) can earn citations and referral traffic but means your content trains and feeds models. Blocking GPTBot, ClaudeBot, and similar in robots.txt protects your content but removes you from AI answers. Decide first, then make robots.txt and any llms.txt agree with that choice.