Retrieval metrics in the LLM era

Pipeline-split lab: Module 5 closed with a judged support corpus, so Module 6 opens on the other production shape — a docs assistant that retrieves passages and then writes from them. An end-to-end score of 66.7% decomposes into 4 retrieval failures, 6 generator failures over a complete context and 20 correct answers, and a perfect generator would still read 86.7%. One retriever reads 85.6% recall@5 against a strict gold set, 91.7% against the passages the answer read and 46.7% against the whole doc. A chunker that halves every passage takes recall@5 from 85.6% to 83.3% while precision@5 rises 29.3% to 34.7% and distinct passages per window fall 5.0 to 4.57; a doc landing costs recall with no twins and no MRR move; a title-weighted scorer lifts MRR 0.9233 to 0.95 with the answer score unchanged; and a prompt rewrite drops the answers to 56.7% with every retrieval number byte-identical. Raising k from 5 to 10 buys 3 complete contexts and 2 answers for 2.06x the context, and one question shares no term with its gold passage, so no k reaches it.

This Evals Engineering lesson is part of a paid plan. Sign in with a plan that includes this course to read it.