How Do AI Search Engines Actually Read Your Page?

How do AI search engines actually read your page?

Not as a page. Retrieval systems cut documents into chunks of a few hundred tokens, turn each chunk into a vector, and match the question against chunks rather than whole articles. The unit that gets retrieved and cited is a passage, and most pages are written as though the unit is the page.

That gap is the whole problem. A brilliant article whose key claim depends on three paragraphs of earlier context will lose to a mediocre one whose key claim stands alone, because only one of them survives being cut into pieces.

We should be honest about the limits of what anyone knows here. No answer engine publishes its exact chunking rules. What is published, by the people who build retrieval systems and by Google, is enough to work from, and that is what this piece uses.

What is chunking, in plain terms?

Chunking is splitting a document into smaller pieces before storing it for search. Anthropic's write up of contextual retrieval, published on September 19, 2024, describes the standard approach as breaking a corpus into smaller chunks of text, usually no more than a few hundred tokens. Its own cost worked example assumes 800 token chunks.

A few hundred tokens is roughly two to four paragraphs of ordinary prose. So the practical question for a writer is straightforward: does each two to four paragraph stretch of your page still make sense if someone reads only that?

The system then retrieves a set of the best matching chunks, not the best matching page. Anthropic's experiments used the top 20 chunks as their standard setup, and also tested 5 and 10.

Why does chunking break so many good pages?

Because ordinary writing carries context forward. You introduce a subject in paragraph two and say "it" for the next nine. When chunk seven is lifted out on its own, the subject is gone, and a system deciding whether that chunk answers a question cannot tell what it is about.

Anthropic names this directly as the failure that contextual retrieval was built to fix: chunks lose the context of the surrounding document. Their solution is to add context back to each chunk before embedding it.

Their numbers show how much that matters. On a top 20 chunk retrieval task, the baseline failure rate was 5.7%. Contextual embeddings alone cut it by 35%, to 3.7%. Adding contextual BM25 cut it by 49%, to 2.9%. Adding reranking cut it by 67%, to 1.9%.

Those figures describe a retrieval system fixing the problem on its side. You cannot run their pipeline over someone else's index, but you can write pages that need less of that repair in the first place.

Is this only about AI, or does classic search work this way too?

Google has worked at passage level for years. On October 15, 2020, Google announced that by understanding passages in addition to the relevancy of the overall page, it could find what it called needle in the haystack information, and said the change would improve 7 percent of search queries across all languages as it rolled out globally.

Google was careful that this is ranking rather than indexing. It still indexes the whole page. It can simply rank a specific passage's relevance as part of judging the page.

The same announcement noted that BERT was by then used in almost every English query. The direction has been consistent for six years: less matching of pages to keywords, more understanding of passages against questions.

What does a self contained chunk look like?

It names its subject in full, states one claim, and supports that claim without reaching backwards. In practice it means repeating the noun instead of using a pronoun, and repeating the qualifier instead of assuming it carried over from the section heading.

Compare two openings. "It drops to about 2.9% when you combine both methods" is useless alone. "Combining contextual embeddings with contextual BM25 reduced the top 20 chunk retrieval failure rate to 2.9%" survives being lifted out of context entirely.

The second is slightly more repetitive to read straight through. That is the trade, and for anything you want cited, it is worth making.

How should you structure a page for passage retrieval?

Use question shaped headings and answer each one immediately underneath. A heading that states the question and a first paragraph that answers it in forty to sixty words produces a chunk that is a complete question and answer pair, which is the shape a retrieval system is looking for.

Keep each section to one idea. Two ideas in a section means the chunk boundary can land in the middle of the second one, and you have produced a half thought.

Put the most citable sentence early in the section, not at the end. If the chunk boundary truncates anything, let it truncate the elaboration rather than the claim. This is the same structural logic behind winning featured snippets, which has always been a passage level game.

Where should the facts and numbers go?

In the same sentence as their source and their subject. A statistic in one paragraph and its attribution in the next produces two weak chunks instead of one strong one, and an engine that retrieves the number without the source has every reason not to quote it.

Write the full triple every time: what the number is, who published it, and what it measures. It reads slightly more formally. It also means any single sentence can be lifted and remains true and attributable, which is the entire game.

The same applies to dates. "Last year" is meaningless in a retrieved chunk. The actual year is not.

Does this mean shorter pages?

No, and this is where the advice often goes wrong. Length is neutral. A long page is simply more chunks, and more chunks means more chances to match a question. What matters is whether each chunk stands alone, not how many there are.

The failure mode of long pages is not length. It is dependency, where later sections only make sense after earlier ones. A long page written as a set of independent answers is the strongest thing you can publish for this kind of retrieval.

So the instruction is not to cut. It is to make each section survive on its own, which we treat as a core part of getting cited by answer engines.

How do you test whether your page chunks well?

Do it by hand, because it takes ten minutes. Copy your page into a plain document, cut it into blocks of roughly three paragraphs, and read each block cold without the rest. Mark every block where you cannot tell what the subject is, or where a number has lost its source.

Those marks are your rewrite list. In our experience the first two or three sections of any article pass easily and the middle is where it falls apart, because by then the writer has settled into shorthand.

A second check: paste one middle section alone into an assistant and ask what it is about. If the answer is vague, the chunk is vague, and no amount of page level optimisation fixes that.

What does this change about how you brief writers?

Add one line to the brief: every section must make sense if read alone. That single instruction does more for retrieval than a keyword list, because it changes sentence construction rather than vocabulary.

Drop instructions that fight it. Requests to vary the terminology, avoid repeating the product name, or keep it flowing all reduce chunk independence. Consistency of terms is a retrieval feature, not a style weakness.

The broader shift is that structure is now part of the writing rather than something applied afterwards, which is also the argument behind covering the full fan out of related questions in one piece.

What is the honest summary?

Retrieval systems work on passages, this is documented by the people who build them and by Google, and pages that depend on running context lose ground to pages that do not. That is the reliable part.

The unreliable part is anyone telling you the exact chunk size a given answer engine uses, because none of them publish it. Treat specific numbers about chunk boundaries with suspicion, including ours, and write for the general shape instead.

We build sites and content systems for B2B companies where being cited is the point, so this is the kind of structural detail we argue about a lot. If you want a second read on whether your best pages survive being cut up, we are happy to look. You can find us at phoenix.studio.