How Do You Make Developer Docs Easy for AI to Quote?

How Do You Make Developer Docs Easy for AI to Quote?

Give machines a clean path to the text. That means an llms.txt index at your docs root, a plain markdown version of every page, one clear answer at the top of each section, and no important content locked behind JavaScript. Most docs sites already have the words. They just hide them badly.

This matters more than it used to. When a developer asks an assistant how to authenticate against your API, the assistant answers from whatever it can parse. If your docs are a heavy single-page app and a competitor ships plain markdown, the competitor becomes the answer.

We build a lot of documentation sites, and the pattern is consistent. The technical work here is small. The hard part is that nobody owns it, so it never gets scheduled.

Why Do Docs Get Cited More Than Marketing Pages?

Because docs are written the way answer engines want to read. A docs page states what a thing is, shows how to use it, and lists what can go wrong. There is no narrative, no brand voice to wade through, and no persuasion. The passage an AI needs is usually the whole section.

Marketing pages do the opposite. They open with a promise, build tension, and put the specifics near the bottom if at all. That structure works on a human who has already decided to read. It gives a machine nothing liftable in the first 200 words.

There is a second reason. Docs answer the exact questions people type. How do I rotate an API key. What does error 429 mean. Those are queries, verbatim. Your homepage answers a question nobody types.

What Is llms.txt and How Do You Write One?

It is a markdown index file that points AI systems at your best content. Jeremy Howard proposed it in September 2024, and llmstxt.org published a version 2 update in August 2026. The idea is simple: web pages are built for people, and turning that HTML back into clean text is difficult and imprecise, so give agents a curated map instead.

The format is strict in only one place. The spec allows an optional byte-order mark, then requires an H1 with the name of the project or site. That H1 is the only required section. After it you may add a blockquote with a short summary, then any markdown sections that are not headings, then any number of H2 sections containing file lists.

The convention worth copying is the Optional section. Links placed under a heading named Optional are the ones an agent can skip when its context window is tight. You are telling the machine what to drop first, which is a kindness no other file format offers.

What Does a Real llms.txt Look Like?

Anthropic publishes one for its developer documentation, and it is a useful model. The file at platform.claude.com/llms.txt is plain markdown. It opens with an H1 reading Anthropic Developer Documentation, followed by a short line describing what the file covers, then sections for each area of the docs.

Underneath, links are grouped as bullet lists under headings such as Messages, Admin, Best practices, and API reference. Each entry is a link with a title and a short description after a dash. The English section alone runs to several hundred links, and the file also indexes versions in other languages with their page counts.

Two details are worth stealing. Every link points at a markdown URL rather than an HTML one, and every link carries a one-line description. The description is what lets an agent choose a page without fetching it first.

Should You Serve a Markdown Version of Every Page?

Yes, and it is the single highest-value change on this list. The llms.txt spec defines the convention: for a page at some URL ending in .html, serve the same content at that URL with .md appended or the extension swapped. For a path with no filename, serve index.html.md or index.md.

The reason is bandwidth and fidelity. A markdown mirror strips the navigation, the cookie banner, the search widget, and the analytics. What is left is the thing the reader wanted. An agent that fetches the markdown gets a cleaner signal and burns far less of its context window doing it.

Most modern docs frameworks can emit this. If you are building a docs site on Webflow rather than a dedicated platform, the approach differs, and we covered the trade-offs in our guide to building a documentation site in Webflow.

What Is llms-full.txt and Do You Need It?

It is the same idea taken to its conclusion: a single file containing the full text of your documentation rather than links to it. An agent fetches that one URL and has everything, with no crawl to run and no follow-up requests to make. The trade is size, because everything means everything.

Whether you need it depends on size. For a small product with 30 docs pages, a full file is genuinely useful and cheap to generate. For a large documentation set, the file gets too big to be loaded into any reasonable context window, and the index version is more practical.

Our honest read is that llms.txt is the one to ship first. The index is small, it is easy to keep current, and it degrades gracefully. A stale llms-full.txt is worse than none, because it serves confidently wrong content with no date on it.

How Should You Structure a Docs Page So AI Can Extract It?

Lead every section with the answer. A heading phrased as a question, then 40 to 60 words that answer it completely, then the detail. That first block should make sense to someone who never read the paragraph above it, because that is exactly how it will be quoted.

Then be specific about names. Use the real parameter name, the real error code, the real endpoint. Vague docs are unciteable docs, because there is nothing in them that uniquely identifies your product as the source of the answer.

Finally, put version and date information in the text, not only in a sidebar widget. If a page applies to version 3 of your API, the words version 3 should appear in the body. An assistant reading a markdown mirror never sees your sidebar.

Does Schema Markup Help Documentation Pages?

It helps less than people assume, and it is still worth doing. Google's structured data guidance for articles supports three types: Article, NewsArticle, and BlogPosting. Google states there are no required properties and tells you to add the ones that apply to your content.

The properties Google recommends are the useful ones anyway. Author, including author name and author URL, plus datePublished, dateModified, headline, and image. For documentation, dateModified is the one that earns its place, because it tells every system reading the page how current the instructions are.

Notably, Google's article guidance does not list TechArticle among the types it supports for this feature. Use BlogPosting or Article for docs-adjacent content and spend the saved effort on the page text. We go deeper on this in our guide to schema markup for SEO.

How Do You Check That AI Crawlers Can Actually Read Your Docs?

Fetch your own pages the way a bot would. Request a docs URL with JavaScript disabled and read what comes back. If the body is an empty div and a script tag, every answer engine sees the same empty div. That single test catches more problems than any audit tool.

Then check robots.txt for rules that block GPTBot, ClaudeBot, PerplexityBot, or Google-Extended. Docs are often served from a subdomain with its own robots file that nobody has reviewed since launch, and a blanket disallow on a staging rule can quietly cover production.

Last, check your server logs rather than your config. A rule that permits a crawler is not evidence the crawler came. If you are also publishing instructions for agents that act on your site, our piece on writing agent instructions for your website covers that side of it.

What Should a Small Team Ship First?

In order: a markdown mirror of every docs page, an llms.txt index pointing at those markdown URLs with a one-line description each, and a question-shaped heading with a direct answer at the top of your twenty most-visited pages. That is a week of work for one engineer and it changes how machines see your product.

Skip the rest until those three are done. Schema, full-text files, and crawler negotiation are all real, but they are refinements on top of content that a machine can already read cleanly. Most teams do them in the opposite order and wonder why nothing moved.

If you want help working out what your docs look like to a machine, we are happy to take a look. That kind of audit is a normal week for us at phoenix.studio, and it usually turns up two or three fixes worth more than a quarter of new writing.