Home/Benchmarks

Benchmark report · November 2026

The AI Responsive Document Review Benchmark

A test of how well language models apply a document review protocol, and what each one costs per 1,000 documents.

36 models evaluated on their ability to identify responsive documents at three different richness levels.

Each model labeled Jeb Bush administration emails from the TREC 2016 Total Recall collection[1] as responsive or not, one email at a time, using the same nine-issue review protocol.[2] 35 models ran all three panels of 1,000 emails; DeepSeek V4 Pro ran only the 20% panel. Models on a given panel saw the same emails. The panels differ in richness, the share of emails that are responsive: 20%, 10%, and 5%. Real reviews land at levels like these, so testing all three shows how each model holds up as responsive documents get rarer, instead of resting on a single data point. We score each model on recall (the share of responsive emails it found), precision (the share of its responsive calls that were right), and F1, which combines the two on a 0–1 scale.

Abstract & summary

A close race and a precision problem every model shares

Claude Opus 5.5 had the highest average F1, 0.571, and led the 10% panel. The field behind it is close: the ten highest-scoring models fall within 0.035 F1 of one another, and six of them cost under $3 per 1,000 documents. Claude Haiku 5.5 averaged 0.545 at a recorded $0.48 per 1,000, about one sixty-fifth of Opus 5.5’s estimated $31.51 API cost.

The open-weight gpt-oss-20b, at $0.24 per 1,000, had the highest F1 on the sparsest panel, 0.478.

Every model scored lower as responsive documents became rarer, and the loss came from precision. Each top-ten model wrongly flagged a nearly constant share of non-responsive documents across the panels, while its recall rose. At 5% richness, those false positives outnumbered the responsive documents found: every top-ten model sent more non-responsive than responsive documents to the responsive pile. Redgrave LLP’s 2026 study modeled this base-rate effect with a formula;[3] these three panels measure it directly.[4]

01 / Compare the results

The top 10, across three levels of richness

02 / Follow the change

Who holds up as responsive documents get rarer?

Each line follows one model from 200 responsive documents in 1,000 to just 50. The dashed line is a reviewer that marks every document responsive, the baseline a model has to beat to filter anything out. Switch to precision or recall to see what sits behind the F1 score.

F1 across the three panels

[5]

03 / See why precision falls

Same error rate, fewer documents to find

Each grid is the responsive pile one model produced on a panel: every email it flagged, one dot each. Orange emails were responsive. Blue (assessor-rejected) and pale blue (unjudged) were not. As richness falls, the pile shrinks only a little while its responsive share collapses.

What one model sent to the responsive pile

  • Responsive
  • Not responsive: assessor-rejected
  • Not responsive: unjudged background

04 / What to take away

The best model depends on the task you give it

1

Precision, not recall, is what falls.

2

A small model is a serious contender.

[6]

3

General capability does not guarantee following the protocol.

Document labeling asks a model to apply a protocol’s inclusions and exclusions consistently. Broader knowledge and more reasoning do not automatically keep those decisions faithful to the instructions.[7]

Some smaller models do well because they do less. They add fewer assumptions beyond the protocol and leave less room to reason their way into a connection the instructions exclude. A more capable model can find a plausible connection that is still the wrong answer under the protocol.

05 / Compare cost and performance

Close scores, very different prices

For nearly the same average F1, the top ten’s costs run from $0.21 (MiMo v2.6 Flash) to $31.51 (Claude Opus 5.5) per 1,000 documents, about 150-fold.[8]

The top ten: average F1 and cost per 1,000 documents

  • Recorded run charge
  • API list-price estimate

06 / Review procedure

How the protocol was applied

1. Define relevance before the run.

The protocol supplies issue definitions, include and exclude lists, coding rules, and the criteria for key documents.

2. Read each email independently.

Each call sends the protocol and one email’s metadata and body, with no richness figure, running tally, neighboring documents, or thread context, so every decision rests on that email alone.

3. Compare decisions with the benchmark labels.

A document counts as responsive if it is responsive to any of the nine issues. Scoring tracks F1, precision, recall, and false positives on assessor-rejected and unjudged background documents separately.

4. Make the non-responsive half hard.

Non-responsive documents in each panel are split equally between assessor-rejected and unjudged background emails. Half are emails a TREC assessor reviewed and rejected, a much larger share of near misses than a typical review set contains, so each panel is harder than usual.

Read the full methods, serving routes, and nine issue definitions

07 / Data and reproduction

Run the same test yourself

We publish the protocol, the prompts, and the 3,000 document numbers. We do not publish the emails themselves.

The documents

The 3,000 panel emails come from the athome4 collection (290,099 emails with assessor judgments on 34 topics). The nine review issues are topics 401–409.

NIST TREC Total Recall data page

Why the emails aren’t here

NIST licenses the Total Recall collection under a usage agreement. It allows research use only, limits access to people the licensed organization lists, and prohibits reproducing or distributing the emails beyond small excerpts. That rules out sharing them here. Decoder Ring Labs applied on June 30, 2026, and NIST approved the agreement on July 1, 2026. Anyone can apply the same way.

NIST organizational usage agreement (PDF)

What we publish

With a licensed copy of the collection, the published document IDs (the collection’s DOCNOs, sorted by ID) let you rebuild each panel.

Download the 3,000 document numbers (CSV)
Read the exact protocol and prompts

Thank you

We thank NIST and the organizers of the TREC Total Recall track for the collection and its relevance judgments, and NIST for approving our access.

Full results for all 36 displayed models

Downloads

Take the results with you

Footnotes
  1. Documents and assessor judgments come from the TREC 2016 Total Recall track, athome4 collection; see Maura R. Grossman, Gordon V. Cormack & Adam Roegiest, “TREC 2016 Total Recall Track Overview,” Proceedings of TREC 2016. The panels were drawn with seed 20260803. See Data and reproduction. Back ↑
  2. Protocol version LB-1.0 (SHA-256 ac07529d…). Where the protocol was placed differed by serving route. Read the exact protocol, prompt, document format, and placement by route. Back ↑
  3. Robert D. Keeling, Ray Mangum, F. Eli Nelson & Kevin A. Reiss, Generative AI for Complex Document Review: A Comparative Evaluation Benchmarked Against Active Learning and an Independent Expert Reviewer, Redgrave LLP Working Paper 2026-01 (June 2026), §5.2, which projects precision across richness while holding recall and the false-positive rate constant. In our panels the false-positive rate held but recall rose. Full report. Back ↑
  4. Each model ran once. Documents with no usable answer after retries were left out of scoring; full results show the counts. Costs use recorded charges for 19 models: 17 OpenRouter benchmark batches matched by model version and run dates, plus recorded totals for Claude Haiku 5.5 and Clef Flash. Matched batches include billed retries and setup checks; later account activity is excluded. Totals are normalized to 1,000 documents from the fixed panel populations. Other priced rows retain dated API list-price estimates; subscription-agent estimates, including Opus 5.5, are API equivalents rather than amounts paid. Scores treat unjudged background documents as non-responsive, over-sample assessor-rejected documents, and do not predict performance in a live review. Back to summary ↑
  5. These are three disjoint panels, so document mix changes along with richness. Lines connect panel scores; they do not isolate the effect of richness alone. All models on a given panel saw the same documents. Back to chart ↑
  6. Back to takeaways ↑
  7. Back to takeaways ↑
  8. Models run through subscription agents, such as Opus 5.5, show estimated API list-price costs as of November 15, 2026, rather than amounts paid. ≥ marks a minimum estimate. Back to costs ↑