llms.txt vs robots.txt — two files, two jobs.

At a glance

What robots.txt actually does

The Robots Exclusion Protocol was formalised in RFC 9309. The semantics are narrow: a compliant crawler reads the file before fetching anything else on the host, matches its User-agent against the listed groups, and treats any URL covered by Disallow: as off-limits. Allow: lifts a more specific path out of a broader Disallow:.

A minimal AI-aware robots.txt looks like this:

User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /private/

User-agent: anthropic-ai
Disallow: /private/

Sitemap: https://www.example.com/sitemap.xml

Three things that robots.txt cannot do — and that teams routinely ask of it:

• It cannot tell engines where the good content is. The file lists what is off-limits, not what is preferred. Engines still have to discover preferred URLs through other means (sitemap, on-page links, external citations).
• It cannot pass a Markdown alternate. robots.txt has no concept of content format. If you serve /page/ as HTML and /page/index.md as Markdown, neither is mentioned to the bot — the bot has to find both via crawling.
• It cannot block a non-compliant bot. The whole protocol is voluntary. If a bot ignores robots.txt, the file does nothing.

What llms.txt actually does

llms.txt was proposed in 2024 by Jeremy Howard (Answer.AI) as an AI-specific analogue to sitemap.xml with two important differences. First, it is Markdown rather than XML, which matches how LLMs prefer to ingest content. Second, it is editorial — it lists the content the site wants the AI to read, not every URL on the host.

The shape is a Markdown document with three load-bearing sections:

# Eastbound

> Eastbound is a Hong Kong-based China AI visibility consultancy.

## Links

- [Homepage](https://www.eastbound.ai/)
- [China AI visibility (pillar)](https://www.eastbound.ai/china-ai-visibility/)
- [AI visibility audit](https://www.eastbound.ai/ai-visibility-audit/)

## Markdown alternates

- [Homepage (Markdown)](https://www.eastbound.ai/index.md)
- [China AI visibility (Markdown)](https://www.eastbound.ai/china-ai-visibility/index.md)
- [AI visibility audit (Markdown)](https://www.eastbound.ai/ai-visibility-audit/index.md)

The format is simple on purpose. It is human-readable, AI-readable without parsing infrastructure, and lets the site editorially curate which content matters. Engines that look for /llms.txt can then prefer the listed Markdown alternates over rendered HTML — which means cleaner extraction with no navigation noise, no boilerplate, no advertising clutter.

What llms.txt cannot do:

• It is not a block list. Listing fewer URLs in llms.txt does not stop engines from crawling the rest. If you want a URL excluded, that is a robots.txt entry.
• It is not a search-engine signal. Google, Bing and Baidu do not currently use llms.txt for indexing. Whether they will is open.
• It is not universally honoured. Engine support is partial. Many AI products read it; some do not. Treat it as upside, not as a guarantee.

Where the two files do not overlap (and where teams confuse them)

A common confusion: a team blocks GPTBot in robots.txt for training-opt-out reasons, then ships an llms.txt assuming it overrides the block for citation purposes. It does not. llms.txt has no enforcement layer; it is a hint. robots.txt is the gate. If GPTBot is disallowed, OpenAI's training crawler will not fetch anything regardless of what llms.txt says — and because OpenAI keeps separate user-agents for training (GPTBot) and search (OAI-SearchBot), the only way to be opted out of training while still appearing in ChatGPT search results is to allow OAI-SearchBot in robots.txt and disallow GPTBot.

Another confusion: assuming robots.txt: Sitemap: directive and llms.txt are equivalent because both list URLs. They are not. The Sitemap: directive points at sitemap.xml, which is an exhaustive index designed for search-engine crawl scheduling. llms.txt is editorial — the curated subset of content you want AI engines to prefer, with Markdown alternates listed alongside. The two coexist; they don't replace each other.

The minimum AI-ready configuration

For a site that wants to be cited by AI engines without being trained on, the typical baseline is:

1. /robots.txt — allow general crawling at User-agent: *; allow named search-and-citation bots (Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended if you are comfortable with Google AIO inclusion); disallow named training bots (GPTBot, anthropic-ai, CCBot) if your editorial position is opt-out-of-training; include Sitemap: directive.
2. /sitemap.xml — exhaustive, machine-readable URL list with <lastmod> and <priority>. Submit this URL to Google Search Console and Bing Webmaster Tools.
3. /llms.txt — editorial index of the pages you want AI engines to prefer, with a parallel "Markdown alternates" section listing the .md equivalents.
4. Per-page Markdown alternates — every page in llms.txt ships a corresponding .md file at the canonical path (e.g. /foo/index.html alongside /foo/index.md) and references it in HTML head with <link rel="alternate" type="text/markdown" href="/foo/index.md" />.
5. /llms-full.txt (optional) — a single-file long-form Markdown bundle of the site's most valuable content, useful for engines that prefer one fetch over many. Eastbound ships one alongside /llms.txt.

Each of these is a baseline-level intervention — measured improvement in crawler fetch behaviour is observable, not speculative. None of them are conversion levers in isolation: shipping these signals is necessary infrastructure, not a result. See AI crawler readiness for the full configuration guide and AI crawler blocking mistakes for the common failure modes.

Format gotchas worth knowing

Case sensitivity

robots.txt filenames are case-sensitive on most webservers. The file must be robots.txt at the root — not Robots.txt, not robot.txt. Same applies to llms.txt. Bots fetch the exact path; a typo means the file is silently absent.

Path placement

Both files belong at the host root, not in subdirectories. /blog/llms.txt is invisible to engines; only /llms.txt is checked. Per-host means per-subdomain too: blog.example.com needs its own /llms.txt and /robots.txt independent of www.example.com.

Content-Type

Both files should be served with Content-Type: text/plain (for robots.txt) or text/plain / text/markdown (for llms.txt — engines accept either). Some bots refuse files served as application/octet-stream. If you deploy from S3 or another object store, set the content type explicitly on upload.

Canonicalisation

If your site lives at www.example.com with apex example.com 301-redirecting, place robots.txt and llms.txt on the canonical host (www.example.com). Bots that follow redirects to fetch example.com/robots.txt generally respect the redirected file, but some implementations cache the apex 404 instead. The safe pattern is to serve the file from both apex and www.