Calibrate the judge: confusion matrix, TPR and TNR above 80 per cent

Calibration lab: L5.2 shipped a judge whose every verdict agreed with its label. Against 19 expert-labelled rows it reads 84.2% — and a judge that answers pass to everything reads 84.2% too, because the gold set is 16 passes to 3 failures. Both rates, or neither: the inherited prompt catches 2 of 3 failures at 87.5% specificity, v4 clears specificity at 100.0% and drops recall to 33.3%, and only the fifth rubric clears 100.0/100.0. The error count runs 3/1/1/2/0, one iteration buys nothing and one costs a failure, and 3 failures means the 80% recall floor is a demand for 3 of 3.

This Evals Engineering lesson is part of a paid plan. Sign in with a plan that includes this course to read it.