Workflow & Validation
Validation Sample
Definition
A random sample reviewed to estimate the quality of a completed review, such as its recall.
Why it matters
It is the evidence behind a claim that the review was reasonable.
In practice
A validation sample is a random sample reviewed after the workflow is finished, to measure how well it did. It is the evidence behind a recall estimate.
Two designs are common: an elusion sample drawn only from the documents set aside, combined with the count of responsive documents found; or a stratified sample drawn from both the reviewed and the set-aside documents, which estimates recall directly.
The sample has to be random, and reviewers should code it without seeing the model scores, so the estimate is not nudged by the result it is checking.
Worked example
Review found 12,000 responsive documents and set aside 300,000. A validation sample of 1,000 set-aside documents finds 5 responsive: 0.5%, or about 1,500 missed. Estimated recall is 12,000 ÷ 13,500, about 88.9%.
Illustrative numbers, not from a real matter.
Compare with
- Control Set: A control set is drawn before training; a validation sample is drawn after review.
Work it out
- Elusion & Recall calculator · What did we leave behind, and what recall can we defend?
- Control Recall calculator · Does a known control set confirm our recall?