llms.txt: The Standard Every Website Needs in 2026

Most website owners know exactly what robots.txt does. Almost none of them have heard of llms.txt.

That's a problem in 2026, when AI systems like Perplexity and ChatGPT are actively crawling the web and making citation decisions that determine which sites get visibility and which get ignored.

llms.txt is a plain-text file that tells AI crawlers which content matters most on your site. It takes 30 minutes to implement. Most of your competitors haven't done it yet.

What llms.txt Is and Where It Came From

llms.txt is a proposed standard, not a W3C specification or an official protocol. Jeremy Howard, the researcher behind fast.ai and ULMFiT, published the proposal in September 2024 with a simple premise: AI systems need better guidance about what to read on a website, and a simple text file at a predictable URL is the cleanest way to provide it.

The file lives at the root of your domain, at /llms.txt, the same predictable location as robots.txt and sitemap.xml. Any AI crawler that supports the standard checks for it when visiting your site.

The format is Markdown. An H1 header gives the site name and a one-line description. Optional H2 sections organize links by category (Documentation, Blog, API Reference, etc.). Each bullet point is a Markdown link with a short description of what that page contains. That's the entire spec.

Here's a real example of a complete, well-structured llms.txt file:

# Ranking Lens

> AI visibility and GEO optimization tools for websites of all sizes.

Ranking Lens helps SEOs, content teams, and developers understand and improve
how their content is cited by ChatGPT, Perplexity, Google AI Overviews, and Claude.

## Guides

- [GEO Optimization Guide](/geo-optimization-ai-visibility): Complete guide to Generative Engine Optimization, covering factual density, structured formatting, and the 3 core signals AI systems use for citation.
- [How to Appear in AI Answers](/how-to-appear-in-ai-answers): Step-by-step process for getting cited in ChatGPT, Perplexity, and Google AI Overviews, including a free AI visibility audit.
- [llms.txt Implementation Guide](/llms-txt-guide): How to create and implement llms.txt for Next.js, WordPress, and static sites.

## Tools

- [Free GEO Analysis](https://rankinglens.com): Instant AI visibility audit showing your GEO score, E-E-A-T signals, and structured data coverage.
- [GEO Basics Guide](https://rankinglens.com/geo-basics-guide): Introduction to GEO fundamentals including llms.txt, content structure, and AI citation prerequisites.

## Optional

- [Sitemap](https://rankinglens.com/sitemap.xml)

The format intentionally simple. No JSON, no XML, no schema to validate against. Just Markdown that both humans and machines can parse at a glance.

How llms.txt Differs from robots.txt

These two files are easy to confuse because they sit in the same location and both address the relationship between your site and bots. But they do fundamentally different things.

robots.txt is a gate. It controls whether a crawler is allowed to visit a URL at all. If you Disallow a path in robots.txt, a compliant bot won't fetch it.

llms.txt is a tour guide. It can't block access to anything. What it does is say, clearly and in plain language, "these are our 10 most important pages, and here's why each one matters."

For GEO optimization specifically, llms.txt is the more impactful file. Most sites don't want to block AI crawlers. They want to direct AI attention toward their best content.

llms.txt vs llms-full.txt: Which One to Create

The standard comes in two variants, and understanding both matters if you're serious about AI visibility.

llms.txt is the index file. It contains links and short descriptions. An AI crawler reads it to know which URLs to prioritize, then fetches those URLs to read the actual content.

llms-full.txt embeds the full text of your key pages directly in the file. No follow-on crawling required. The crawler gets everything in one request.

When to use each one depends on your situation. If your key content is freely accessible, fast-loading, and server-rendered, the standard llms.txt works perfectly. If your site has content behind JavaScript rendering, login walls, or complex layouts that confuse AI parsers, llms-full.txt is the more reliable option because you control exactly what the AI sees.

The size trade-off is real. A clean llms.txt with 15 links might be 2 KB. An llms-full.txt with the full text of those same 15 pages is typically 500 KB to 2 MB. That's not a problem for crawlers, but it's worth keeping the content focused. Include your 10 to 15 genuinely highest-value pages, not everything.

The ideal setup for most sites: create both. Serve llms.txt as the lightweight default. Include a link to llms-full.txt at the bottom for crawlers that want the deep version.

Which AI Systems Actually Read llms.txt

Honest answer: fewer than the hype suggests, but the number is growing.

Perplexity has publicly confirmed support. Their crawler reads llms.txt during site visits and uses it to weight content priority in their index. For GEO purposes, this is the most important confirmation because Perplexity is one of the top-cited AI search platforms and its citation behavior is explicit and trackable.

OpenAI's GPTBot and ChatGPT's browsing feature haven't confirmed active llms.txt support as of March 2026. Their crawlers don't block the file and will fetch it if present, but whether it influences ChatGPT's citation decisions is unconfirmed. The absence of a "no" is not a "yes" here.

Anthropic's ClaudeBot is in the same position: technically able to read the file, no public confirmation of active support.

Google has said nothing about llms.txt in relation to Googlebot or Google AI Overviews as of this writing.

Several smaller AI tools and developer-built assistants have adopted the standard, particularly in the developer tooling space where Jeremy Howard's fast.ai audience has strong influence.

The practical takeaway is straightforward. Perplexity supports it. That's enough reason to implement it. The file is trivially cheap to create, and every additional platform that adopts the standard in the next 12 to 18 months will read the file you've already built.

How to Implement llms.txt for Different Platforms

Next.js (App Router)

The cleanest approach is to place a static llms.txt file in your public/ directory. Next.js serves everything in public/ at the root path with no configuration required.

/public/llms.txt

That's it. Your file is immediately available at yourdomain.com/llms.txt.

If you want a dynamic file that auto-generates from your article metadata, create a route handler:

// app/llms.txt/route.ts
import { getAllArticles } from '@/lib/articles';

export async function GET() {
  const articles = await getAllArticles();

  const lines = [
    '# Your Site Name',
    '',
    '> One-sentence description of what your site covers.',
    '',
    '## Blog',
    '',
    ...articles.map(a => `- [${a.title}](/${a.slug}): ${a.description}`),
  ];

  return new Response(lines.join('\n'), {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
  });
}

This approach keeps your llms.txt current automatically as you publish new articles.

WordPress

Three reliable options:

Upload a plain llms.txt file directly to your server's root directory via FTP or your hosting control panel's file manager. No plugins needed. The file serves immediately at yourdomain.com/llms.txt.

Use a plugin. The "LLMs.txt for WordPress" plugin (free, on the WordPress plugin directory) generates the file automatically from your published posts and pages. Worth installing if you publish frequently and don't want to maintain the file manually.

Add a custom rewrite rule in functions.php if you need programmatic control without a plugin.

Static Sites (Eleventy, Hugo, Jekyll, Astro)

Create an llms.txt file in your site's root content directory. Every static site generator treats root-level files as pass-through assets that serve at the root URL. No build configuration needed.

Why llms.txt Improves Your GEO Citation Rate

The connection between llms.txt and GEO performance isn't magic. It's prioritization.

Without llms.txt, an AI crawler visiting your site has to infer which pages matter most from structural signals: internal link count, crawl depth, sitemap priority tags, and page authority metrics. These signals are imperfect. A deeply nested page that's genuinely your most comprehensive piece of content might score lower than a shallow, frequently-linked page that's mostly boilerplate.

With llms.txt, you're directly telling the crawler: "These 10 URLs are our most complete, most authoritative pages on these topics." That signal is unambiguous.

For Perplexity specifically, which has confirmed it reads the file, this translates directly into which of your pages get indexed with higher priority weighting. Higher priority pages get surfaced more often in Perplexity's citation list when users ask relevant questions.

There's a secondary effect too. Writing a good llms.txt forces you to think clearly about which pages actually represent your site's expertise. If you can't write a one-line description that explains what a page covers and why it's valuable, that's a signal the page might need work. The process of creating llms.txt often surfaces content gaps and weak pages you didn't know you had.

For a deeper look at the full GEO picture, including factual density, FAQ structure, and structured data requirements, see our GEO optimization guide.

Step-by-Step: Create Your llms.txt Right Now

This takes 30 minutes or less.

Step 1. Identify your 8 to 15 most important pages. Don't include everything. Focus on the pages that best represent your expertise, the ones you'd want an AI system to read first if it could only read a handful.

Step 2. Group them into 2 to 4 logical categories that match how your site is organized. "Guides", "Tools", "Documentation", "Case Studies" are all reasonable category names.

Step 3. Write one sentence per page explaining what it covers and why it's useful. Be specific. "Our complete guide to LCP optimization" is better than "a blog post about performance."

Step 4. Assemble the file using the Markdown format shown earlier. H1 for site name and description, H2 for each category, bullet points with Markdown links.

Step 5. Place the file at the root of your domain using the method appropriate for your platform (see the platform instructions above).

Step 6. Verify it's live by visiting yourdomain.com/llms.txt in a browser. You should see plain text, not HTML.

Step 7. Test your overall GEO readiness using Ranking Lens. The free analysis checks whether your llms.txt is detected, structured correctly, and accessible to crawlers, alongside your full AI visibility score.

That's the entire process. 7 steps, 30 minutes, and your site has a signal that your competitors almost certainly don't have yet.

Why Most Sites Haven't Done This Yet

Honestly, the main reason llms.txt isn't universal yet is timing. The proposal came out in late 2024. Most SEOs were still focused on Google SGE and traditional ranking signals through 2025. The AI search landscape moved faster than the advice ecosystem could track.

The second reason is that "proposed standard" signals fragility to cautious practitioners. If OpenAI, Google, and Anthropic don't officially adopt it, does it matter?

It already matters for Perplexity. That's one of the 4 major AI search platforms right now. And the history of web standards suggests that once one major player adopts a useful convention, others follow. robots.txt was a single site's informal proposal in 1994. Google adopted it, and now it's a universal standard.

llms.txt is at that inflection point. The sites that implement it now get the early advantage on platforms that already support it and will be positioned automatically when others adopt it.

If you're serious about AI visibility in 2026, this is one of the fastest, lowest-effort signals you can add to your site. Check our Lovable SEO guide for more context on technical AI visibility foundations if you're building on a JavaScript-heavy stack.

Useful Resources

• Ranking Lens Free GEO Analysis: Check your AI visibility score, llms.txt status, and structured data coverage in one free audit.
• Ranking Lens GEO Basics Guide: Full introduction to GEO optimization including llms.txt implementation, content structure, and citation prerequisites.
• Google Search Console: Monitor AI Overview impressions and track which of your pages are appearing in Google's AI-generated results.
• GEO Optimization Guide: Deep-dive on factual density, structured content, and the 3 signals that drive AI citation rates.