llms.txt for AI search: examples, SEO value, and setup

llms.txt for AI search is worth adding when your site has documentation, product pages, comparison pages, support content, or dense category guides that AI assistants may need to summarize. It will not force ChatGPT, Claude, Perplexity, Gemini, or Google AI Overviews to cite you. The win is simpler: give retrieval systems and AI browsing tools a clean, curated reading list before they guess from your navigation.

The painful version: you publish strong content, but AI answers cite a competitor's cleaner docs page, a stale directory, or a random blog post because your best source page is buried behind menus, scripts, duplicate URLs, and vague titles. An llms.txt file does not replace SEO, schema, or internal links. It reduces the effort required for an AI system, crawler, or agent to find the pages you actually want read.

Use this fast rule: create https://yourdomain.com/llms.txt if you can name 10-50 pages that an AI assistant should read before answering questions about your product, category, pricing, docs, comparisons, or troubleshooting. If you cannot name those pages, fix your content map first.

What is llms.txt?

llms.txt is a Markdown file served at the root of a website that gives language models and AI agents a structured map of the site's most useful pages. The proposed format comes from the public llms.txt specification, which recommends a concise site summary followed by categorized links to important resources.

llms.txt is a curated AI-readable index, not a crawler-control file. It tells an assistant what content is important and where to find it. It does not block bots, grant permission, guarantee indexing, or replace robots.txt, XML sitemaps, structured data, canonical tags, or normal internal links.

A good llms.txt answers four questions:

If you need the broader visibility loop, start with the AI visibility tracking workflow. This guide is the file-level implementation step.

Does llms.txt help AI search visibility?

llms.txt can help AI search visibility by making your most useful source pages easier to discover, summarize, and cite, but it is not a ranking factor you can rely on. Treat it as a low-cost retrieval aid, not a magic GEO switch.

The honest answer is mixed:

The practical benefit is not that every model has promised to consume it. The benefit is that llms.txt makes your preferred source set explicit. Documentation platforms and developer sites have started exposing files such as /llms.txt and /llms-full.txt because AI coding assistants and retrieval tools need compact, readable context. Cloudflare, for example, publishes AI-readable documentation indexes such as developers.cloudflare.com/llms.txt, and X documents both llms.txt and llms-full.txt for its developer docs.

That matters most when AI answers need a source page. For example, a Perplexity-style answer with citations may choose between your pricing page, a competitor comparison, a help article, or a third-party list. The easier your canonical source is to find and quote, the better your odds of owning the cited URL. For the measurement layer, use Perplexity citation monitoring.

What should you put in an llms.txt file?

Put only the pages that should shape an answer about your site, product, category, or expertise. Keep it short enough that an AI assistant can scan it quickly. A bloated llms.txt that lists every tag page, archive, and minor blog post recreates the same discovery problem in a new file.

Use this structure:

# Brand or Site Name

Short description of what the company, product, or site does.

## Core product

- [Product overview](https://example.com/features): What the product does and who it serves.
- [Pricing](https://example.com/pricing): Plans, limits, and buying path.
- [Demo](https://example.com/demo): How prospects can evaluate the product.

## Best source pages

- [Category guide](https://example.com/blog/category-guide): Clear definition and buyer context.
- [Comparison guide](https://example.com/blog/vendor-vs-vendor): Tradeoffs and decision criteria.
- [Troubleshooting guide](https://example.com/blog/problem): How to solve a common failure mode.

## Documentation

- [API reference](https://example.com/docs/api): API endpoints and integration details.
- [Getting started](https://example.com/docs/start): First setup path.

Write the descriptions like source labels, not marketing copy. "Explains plan limits and buyer fit" is useful. "Unlock the future of AI-powered growth" is trash.

For a Tracemetry-style B2B SaaS site, the first version should usually include:

• Homepage, features, pricing, and demo.
• The category pillar, such as what generative engine optimization is.
• Measurement guides, such as AI visibility tracking, AI brand monitoring, and share of voice in AI search.
• Surface-specific guides, such as ChatGPT citation tracking and Perplexity citation monitoring.
• Technical pages, such as schema markup for AI search and content that AI assistants cite.
• Comparison or alternatives pages when they are accurate, current, and fair.

How is llms.txt different from robots.txt, sitemap.xml, and schema?

llms.txt explains what an AI should read. robots.txt controls crawler access. sitemap.xml lists URLs for search engines. Schema describes the meaning of visible page content. You need different files because they solve different jobs.

Use this decision table:

Google's documentation says structured data helps Google understand page content, and Google's AI features guidance says site owners should rely on normal Search controls for how content appears in Search AI experiences. That is the correct mental model: llms.txt is additive. It does not bypass the normal web stack.

Cloudflare's AI Crawl Control docs also separate AI crawler access and content guidance from normal page content. If your goal is permission, crawler control, or pay-per-crawl policy, use the relevant crawler controls. If your goal is "when an AI agent reads us, show it the best sources first," use llms.txt.

What is the best llms.txt example for a SaaS website?

The best SaaS llms.txt example is a short Markdown file with product context, buyer paths, docs, pricing, comparison pages, support pages, and your strongest source-worthy articles. It should be closer to a hand-curated source brief than a full sitemap.

Here is a practical SaaS template:

# Example SaaS

Example SaaS helps B2B marketing teams monitor AI-generated answers, track competitor mentions, and publish content that improves source ownership.

## Product

- [Features](https://example.com/features): Product workflow for monitoring, briefs, drafts, review, publishing, and re-measurement.
- [Pricing](https://example.com/pricing): Plan limits, prompt volumes, and buying path.
- [Demo](https://example.com/demo): Evaluation path for teams that want a guided walkthrough.

## Core guides

- [What is answer engine optimization?](https://example.com/blog/answer-engine-optimization): Category definition and operating model.
- [AI visibility tracking](https://example.com/blog/ai-visibility-tracking): Weekly prompt, mention, citation, and competitor measurement workflow.
- [Content that AI cites](https://example.com/blog/content-that-ai-cites): Page structure and source standards for AI citations.

## Comparisons

- [Example SaaS vs Enterprise Vendor](https://example.com/compare/enterprise-vendor): Fit, pricing, workflow, and tradeoffs.
- [Best AI visibility tools](https://example.com/blog/ai-visibility-tools): Tool comparison for buyers.

## Technical references

- [Schema markup for AI search](https://example.com/blog/schema-markup-for-ai-search): Structured data types and examples.
- [API documentation](https://example.com/docs/api): API endpoints and developer reference.

Keep the file honest. If you include pages that do not clearly answer the description beside them, AI systems and human reviewers learn that your source map is unreliable.

When should you publish llms-full.txt too?

Publish llms-full.txt only when your content is compact, stable, and genuinely useful as one bundled context file. For most marketing sites, llms.txt is enough. For developer documentation, API references, SDK docs, tutorials, and technical manuals, llms-full.txt can help agents ingest the complete docs without crawling dozens of pages.

Use this rule:

If the full file would include outdated documentation, duplicate articles, thin pages, or private-ish support content, do not publish it. A concise llms.txt that points to the best pages is better than a giant context dump that lowers trust.

How do you implement llms.txt without hurting SEO?

Implement llms.txt as a public, static Markdown file at /llms.txt, link only canonical pages, keep descriptions factual, and update it whenever your product, docs, pricing, or core content changes. It should not create duplicate pages or expose URLs that should stay private.

Checklist before publishing:

• Serve it at https://yourdomain.com/llms.txt.
• Use Markdown, not HTML.
• Include the site name, one concise description, and grouped links.
• Link only public canonical URLs.
• Keep each link description under one sentence.
• Do not include staging URLs, admin URLs, draft URLs, or private support links.
• Do not stuff keywords into every description.
• Add the file to your deployment workflow so it stays current.
• Test it with curl https://yourdomain.com/llms.txt.
• Track whether AI citations, AI referrals, or cited URLs change after publishing.

For Next.js, the simplest implementation is a static file in public/llms.txt. For a docs system, generate it from the same source of truth as the docs navigation. For a content-heavy site, regenerate it when the top source pages change.

How do you measure whether llms.txt worked?

Measure llms.txt by watching source ownership, not vibes. Track whether AI surfaces cite better URLs, whether AI crawlers request the file, whether AI referrals land on fewer irrelevant pages, and whether answer accuracy improves for prompts that used to cite stale or wrong sources.

Use this measurement loop:

1. Pick 20-50 prompts where your site should be cited.
2. Record current brand mentions, cited domains, cited URLs, and answer accuracy.
3. Publish llms.txt.
4. Confirm the file is live and fetchable.
5. Re-run the same prompts weekly for 4-6 weeks.
6. Check server logs for requests to /llms.txt.
7. Fix pages where AI systems still cite competitors or stale URLs.

Do not expect instant movement. AI surfaces recrawl at different speeds, use different retrieval systems, and may ignore the file entirely for some answer types. The file is one signal in a larger operating loop: source-worthy content, clean schema, canonical URLs, internal links, external proof, and repeated measurement.

Start with the source map, then measure

Add llms.txt if your site has clear source pages that AI assistants should read. Keep it short, honest, and current. Then measure whether your cited URLs improve instead of pretending the file itself is the outcome.

Run a free Tracemetry audit to see which AI prompts already mention or cite your site. If the gaps matter, use Tracemetry Pro to track prompts weekly, generate source-grounded briefs, publish the missing pages, and re-measure whether AI answers start using the right sources.

For the surrounding execution work, pair this with schema markup for AI search, content that AI assistants cite, and the generative engine optimization audit.

FAQ

What is llms.txt? llms.txt is a Markdown file at the root of a website that gives AI assistants and agents a curated map of the site's most useful public pages. It usually includes a site summary and grouped links to docs, guides, product pages, comparison pages, and support resources.

Does llms.txt improve AI search rankings? llms.txt can make your best source pages easier to discover and summarize, but it does not guarantee AI search rankings, mentions, or citations. Treat it as a retrieval aid and measure results through AI visibility tracking, citation tracking, and server logs.

Is llms.txt the same as robots.txt? No. robots.txt gives crawler access instructions. llms.txt gives AI systems a curated reading list. Use robots.txt for permissions and crawler policy; use llms.txt to point AI readers toward the pages that best explain your site.

What should I include in llms.txt? Include your most important public canonical pages: product overview, pricing, demo, docs, API references, category guides, comparison pages, troubleshooting pages, FAQs, and source-worthy articles. Leave out thin pages, duplicates, archives, staging URLs, and anything private.

Do I need llms-full.txt? Most SaaS marketing sites do not need llms-full.txt. Documentation portals, API references, SDK docs, and technical manuals may benefit from a bundled full-context file if the content is accurate, compact, and updated automatically.