TPEE Toptronic® Prompt Engineering Editor

117. Revisiting Context 4 - Retrieval, Ranking and the Right Chunk

Retrieval is a ranking problem: chunk boundaries, hybrid search, the re-ranker, and why the top result is often the wrong one.

By Jacques Botte, founder of Toptronic®. Last updated 17 October 2026.

The lesson

Retrieval-augmented generation fails for one of two reasons, and they are worth separating. RECALL failure means the passage that answers the question was never retrieved. PRECISION failure means it was retrieved and buried, or crowded out by near-duplicates. A system that returns ten chunks and uses all ten hides both behind the same symptom: a plausible answer that misses. Measure recall and precision separately or you will fix the wrong one.

Chunking decides what can ever be found. A fixed-size chunk cuts sentences and sometimes cuts a table row from its header, and no retriever can recover meaning that the chunk boundary destroyed. Overlap helps at the edges but doubles storage and produces near-duplicate results. The better default for prose is one section per chunk with its heading carried in the text; for tables, one row plus the header; for code, one function plus its signature and doc comment.

Pure embedding search is good at paraphrase and bad at exact tokens - error codes, part numbers, surnames, identifiers. Pure keyword search is the reverse. Hybrid retrieval runs both and fuses the rankings, which is why it is the default worth reaching for: it costs little and it closes the class of failure where a user searches for a string the embedding considered unremarkable.

The re-ranker is the cheapest quality upgrade in the stack. Retrieve thirty candidates cheaply, then score them against the actual question with a small cross-encoder or a model call, and pass only the best few. It fixes precision without asking the index to be clever. What it cannot fix is recall: if the right passage was never a candidate, no re-ranker will find it.

Then measure. Keep a set of real questions with the passage that truly answers each, and score recall at k, precision at k, and whether the final answer cited the right source. Retrieval quality drifts as the corpus grows - a document added yesterday can outrank the one that matters - so the set is a standing instrument, not a one-off experiment.

Check yourself

Question 1: A system returns ten chunks and the answer still misses. What must be measured separately?
  1. Latency and cost
  2. Recall (was it retrieved at all) and precision (was it buried or crowded out) — correct
  3. Tokens and temperature
  4. Chunk size and overlap

Answer: Recall (was it retrieved at all) and precision (was it buried or crowded out)

Both failures look identical from the answer. Recall failure means it was never retrieved; precision failure means it was retrieved and lost among near-duplicates.

Question 2: Why is hybrid retrieval the sensible default?
  1. It is cheaper than embeddings
  2. Embeddings are good at paraphrase and bad at exact tokens; keyword search is the reverse, so fusing them closes a failure class — correct
  3. It removes the need for chunking
  4. It replaces the re-ranker

Answer: Embeddings are good at paraphrase and bad at exact tokens; keyword search is the reverse, so fusing them closes a failure class

Error codes, part numbers and identifiers are exactly what embeddings find unremarkable, and exactly what keyword search nails.

Question 3: What can a re-ranker fix, and what can it not?
  1. It fixes recall, not precision
  2. It fixes precision among the candidates, but cannot recover a passage that was never retrieved — correct
  3. It fixes both
  4. It fixes neither

Answer: It fixes precision among the candidates, but cannot recover a passage that was never retrieved

Retrieve wide and cheaply, re-rank the candidates accurately, then pass the best few. Recall has to be solved upstream.

← Previous lesson · All 134 lessons · Next lesson →

The full course — 134 lessons and 402 quiz questions — ships inside the app. Get TPEE to study it offline.

This lesson is the written summary of a course that ships inside the TPEE prompt editor for Windows - thirteen sections, a 0-100 score before you send anything, and it works with no network connection. The three one-time prices, and a 14-day free trial.