Workflow & Validation
Overturn Rate
Definition
The proportion of coding decisions changed when documents are re-reviewed during quality control.
Why it matters
A high overturn rate signals inconsistent coding that can undermine results.
In practice
The overturn rate is the share of coding decisions that change when a second reviewer checks them during quality control.
The direction matters. Responsive documents wrongly coded not responsive are missed documents and affect recall; the opposite errors cost review time and affect precision.
A high or rising overturn rate usually points to unclear coding guidance, so it is a prompt to clarify the protocol and, in TAR, to fix training decisions before the model learns from them.
Formula
Overturn rate = decisions changed in QC ÷ decisions checked in QC
Worked example
QC re-reviews 500 first-level decisions and changes 35: a 7% overturn rate. Of the 35, 25 were responsive documents coded not responsive, so most errors were misses.
Illustrative numbers, not from a real matter.
Compare with
- Precision: Precision is measured against final decisions; overturn rate measures disagreement between reviewers.