End-to-end RAG harm checks

Harm-drill lab: L6.1 measured the retriever and L6.2 graded the answer claim by claim, and both grade an answer against the documents the system should have had. This lesson grades the assembled system for the failures users suffer. Three suites over the same 48 passages: 16 abstention cases (8 answerable, 8 not) read 62.5% coverage and 50.0% abstention for 56.3% correct against 50.0% for either constant policy; 6 leakage cases read 100% window leak against 83.3% answer leak, with the tenancy filter closing all six; and one release’s 9 changed passages are drilled with 5 questions, so detection is 100% and coverage of the release is 55.6%. All 5 stale answers are grounded in the index that produced them, because the index agrees with the answer and the corpus does not. Harm-weighted the three suites bill $30.00, $200.00 and $52.00 for 7, 5 and 9 failures: by count staleness leads, by bill leakage does at 70.9% of $282.00, and the breakeven leak price is $10.40 against a declared $40.00. The two constant policies cross at 75% answerable, so a balanced suite recommends the policy a 92% mix punishes.

This Evals Engineering lesson is part of a paid plan. Sign in with a plan that includes this course to read it.