The judge panel: agreement, the decision rule and the ticket

Panel lab: L5.5 priced one calibrated judge, so this lesson adds a second reader and a cheap screen and finds a panel is not an average. Majority of three catches 42 of 47 failures at 96.6% — two fewer than its best member, which catches 44 at 98.4%. The two strong judges each catch 44, share 41, and their union catches 47 of 47 for 2 false alarms, because their errors do not overlap. The same 189 votes read four instruments: majority 42 caught / 5 open, unanimity 37 / 14 open, the union 47 / 0 open, all three accusing 43 / 4 cleared. Agreement is not accuracy — the pair that agrees most (52 of 54, kappa 0.899) shares both its errors, and the pair that agrees least (55 of 61, kappa 0.755) shares none. Eight contested rows settled by a person cost $10.5040 and get every row right, against $12.7520 on one judge’s own ten doubts and $78.75 on all 63. And 87.5% specificity is 95.9% precision on the corpus, 72.7% at a 25% queue and 29.6% on a clean 5% queue — the same panel, four numbers, one mix.

This Evals Engineering lesson is part of a paid plan. Sign in with a plan that includes this course to read it.