Workflow & Validation
Control Set
Definition
A fixed reference sample used to compare workflow behavior over time.
Why it matters
Helpful in some workflows, but weak design can mislead.
In practice
A control set is a random sample of the collection, coded by knowledgeable reviewers at the start and kept out of training. Because the model never learns from it, it gives an unbiased view of how the model is performing.
In TAR 1.0 it is used to estimate richness, to decide when training has stabilized, and to estimate recall and precision at a chosen cutoff.
Its weaknesses are size and drift. At low richness a control set contains few responsive documents, so its recall estimates are imprecise, and if new documents join the collection later, the control set no longer represents it.
Formula
Observed recall = control-set responsive documents above the cutoff ÷ all responsive documents in the control set
Worked example
A 2,000-document control set has 240 responsive documents. The model ranks 221 of them above the cutoff, an observed recall of 92.1%. With only 240 responsive documents, the 95% interval runs from roughly 88% to 95%.
Illustrative numbers, not from a real matter.
Compare with
- Training Set: Training documents teach the model. Control documents must never be used for training, or the estimates are inflated.
- Validation Sample: A validation sample is drawn at the end to test the finished result; a control set is drawn at the start.
Work it out
- Control Recall calculator · Does a known control set confirm our recall?