Workflow & Validation

Overturn Rate

Definition

The proportion of coding decisions changed when documents are re-reviewed during quality control.

Why it matters

A high overturn rate signals inconsistent coding that can undermine results.

In practice

The overturn rate is the share of coding decisions that change when a second reviewer checks them during quality control.

The direction matters. Responsive documents wrongly coded not responsive are missed documents and affect recall; the opposite errors cost review time and affect precision.

A high or rising overturn rate usually points to unclear coding guidance, so it is a prompt to clarify the protocol and, in TAR, to fix training decisions before the model learns from them.

Formula

Overturn rate = decisions changed in QC ÷ decisions checked in QC

Worked example

QC re-reviews 500 first-level decisions and changes 35: a 7% overturn rate. Of the 35, 25 were responsive documents coded not responsive, so most errors were misses.

Illustrative numbers, not from a real matter.

Compare with

  • Precision: Precision is measured against final decisions; overturn rate measures disagreement between reviewers.
← Back to glossary