Retrieval, source hierarchy and keeping context clean

Retrieval is how a model answers from documents it was never trained on. The quality of the answer is mostly the quality of what was retrieved.

advanced9 minChecked 2026-09-06

What you will be able to do

Explain retrieval well enough to debug it, and set a source hierarchy that survives contradiction.

Before you start

  • Comfort with the context window and context engineering guides.

How retrieval works

Documents are split into chunks and indexed. A query finds the chunks that look most relevant. Those chunks go into the context. The model answers from them.

Every failure mode lives in that pipeline: chunks split mid-argument, the retrieved passage is topically similar but not the answer, the index is stale, or the right document was never in it.

Debugging by stage

When a retrieval-backed answer is wrong, ask which stage failed. Was the document indexed? Was the right chunk retrieved? Was the chunk sufficient? Did the model use it or override it with its own priors?

Most systems can show you the retrieved passages. Look at them before blaming the model.

Hierarchy and freshness

When sources disagree, something has to win. State the order: the current official document beats a blog post; the signed contract beats the summary; the primary study beats the press release.

Attach dates. A model has no way to know that one of your documents was superseded unless you say so.

What good looks like

  • You can inspect what was retrieved.
  • Your sources carry dates and a stated precedence.
  • The system says when the answer is not in the corpus rather than improvising.

What it cannot do

  • Retrieval reduces fabrication; it does not remove it.
  • Retrieval quality is bounded by the corpus. Missing documents produce confident gaps.

When it goes wrong

Confident answer contradicting your documents.
Check what was retrieved. The model may be answering from training rather than from context. Instruct it to answer only from the supplied sources and to say when it cannot.
Right document, wrong passage.
A chunking problem. Chunk on semantic boundaries rather than fixed character counts, and overlap them.

Privacy and data handling

An index is a copy. Whatever governance applies to the source documents applies to it, including deletion requests.

Sources3 referenced
  1. Claude developer documentation

    Anthropic · official product documentation · link checked 2026-09-06

  2. OpenAI platform documentation

    OpenAI · official product documentation · link checked 2026-09-06

  3. AI Risk Management Framework

    NIST · standards framework · link checked 2026-09-06

Links are checked at build time. If one has moved, tell us.

Written by
BLACK LDN
Last checked
2026-09-06
Content state
editorial

Product behaviour changes. If something here no longer matches the documentation, send a correction.

Read next

Giving it the right information9 min

What context engineering actually means

Prompting is what you ask. Context engineering is what you put in the room before you ask it. The second is where most of the quality lives.

intermediate

Checked 2026-09-06

Next: Draft and review documents without losing your voice