Home/Benchmarks/Protocol and prompts

Benchmark report · November 2026

The protocol and prompts

Everything each model was given, exactly as it was given: the review protocol, the per-document prompt, and the format of each email. The answer key was never shown to any model. Return to the report.

What every model received

Each email was a separate, independent call. No model saw other emails, threads, a running tally, the panel it was labeling, or how many documents were responsive.

1 · The protocol

The full review protocol below (version LB-1.0, about 3,500 tokens): coding rules, the key-document standard, and the nine issue definitions with include and exclude lists.

2 · The task and one email

The per-document prompt below, with one email inserted where it says {document}. For chat models, an email over 60,000 characters was truncated with a marker; Jev read every email in full.

3 · The answer

A grade of 0, 1, or 2 and a confidence for each of the nine issues, plus a one- or two-sentence rationale, as strict JSON. Decision-model routes returned grades through their own formats.

Where the protocol went, by route

The protocol text was byte-identical on every route. Where it was placed differed, because each route offers different slots.

RouteProtocol placed inPrompt and emailNotes
Chat APIs (OpenRouter, Alibaba DashScope)System messageUser messageJSON output mode requested where the endpoint supports it.
Anthropic API (Claude Haiku 5.5)System prompt, cachedUser messageDefault sampling; Claude 5 models reject temperature settings.
Claude Code (Claude Opus 5.5)System promptUser promptFresh session per email, empty read-only folder, no tools. The agent adds about 560 tokens of its own instructions.
Codex (GPT-6.1 Sol)Developer instructionsUser promptFresh session per email, read-only sandbox. The agent adds about 12,000 tokens of its own instructions.
Antigravity (Gemini 4 Hydrogen, Gemini 3.8 Flash)Appended to the system promptUser promptNew session per email; every tool request refused.
Cursor agent (Grok 4.7)Start of the user prompt, between === REVIEW PROTOCOL === and === END PROTOCOL ===Same user prompt, after the protocolCursor has no system-prompt option. It also attaches the account’s own user rules and synced skills.
Decision models (Jev, SemIf, Clef Flash)Their own API fieldsOne email per requestSame protocol text; they return a grade per issue instead of JSON with a rationale.

The per-document prompt

Sent verbatim, with the rendered email in place of {document}.

You are a document reviewer coding one email against the attached review
protocol. Read the protocol carefully, then read the document below and code it
against ALL NINE issues (401–409) independently.

Return ONLY a JSON object, no other text, with exactly this shape:

{
  "docno": "<the document's DOCNO>",
  "issues": {
    "401": {"grade": 0, "confidence": 0.97},
    "402": {"grade": 0, "confidence": 0.99},
    "403": {"grade": 0, "confidence": 0.95},
    "404": {"grade": 0, "confidence": 0.98},
    "405": {"grade": 0, "confidence": 0.99},
    "406": {"grade": 0, "confidence": 0.96},
    "407": {"grade": 1, "confidence": 0.83},
    "408": {"grade": 0, "confidence": 0.99},
    "409": {"grade": 0, "confidence": 0.97}
  },
  "rationale": "one or two sentences explaining any non-zero grades (or the closest call if all zero)"
}

Rules:
- "grade" is the protocol grade: 0 = non-responsive, 1 = responsive,
  2 = key/important. All nine issue keys are required.
- "confidence" is your calibrated probability (0.0–1.0) that the grade-0-vs-
  grade≥1 responsiveness call is correct for that issue.
- Apply the protocol's tie-break rules: doubt between 0 and 1 → 1; doubt
  between 1 and 2 → 1.
- Output must be valid JSON. No markdown fences, no commentary.

DOCUMENT:

{document}

SHA-256 d0327bf308751826f0a99abb2f5f8a17c9b54c4c9927e1a814164242d74af989

How each email was shown

Six fields from the collection, in this order. Nothing else from the email’s context was included.

DOCNO: <six-digit document number>
Date: <sent date>
From: <sender>
To: <recipients>
Subject: <subject>

<email body>

The review protocol

Formatted for reading. The raw file is byte-identical to what every model received; its SHA-256 is stamped into every run record.

SHA-256 ac07529debf3d75c95f55575a2dc382ce6faa495b6436de1fae3d259b897ab10

Document Review Protocol — Jeb Bush Email Collection

Version: LB-1.0 · Reviewers: AI labeling models under benchmark conditions

This protocol governs how a document is judged responsive to each of the nine review issues, and how borderline calls are resolved. Apply it exactly as written; the same standard is applied to every reviewer.


1. How to use this protocol

  1. Treat each of the nine issues as an independent responsiveness call. A single email can be responsive to zero, one, or several issues.
  2. For each document and issue, assign one grade:
    • 0 — Non-responsive: does not concern the issue.
    • 1 — Responsive: concerns the issue (the issue is discussed, requested, decided, forwarded, or materially referenced).
    • 2 — Key / Important: responsive and materially significant to the issue — substantive discussion, a decision or directive, an original report, or a document a case team would want surfaced first.
  3. When in doubt between 0 and 1, code 1 (favor recall — this is a recall-oriented review). When in doubt between 1 and 2, code 1.
  4. Apply the "concerning" standard broadly: a document "concerns" an issue if a reasonable reviewer would understand it to be about, relating to, or materially referencing that issue — not only documents where the issue is the sole subject.

2. General coding rules

These apply across all nine issues.

  • Email families / threads. Judge the document as presented (each record is a single email). If a responsive statement appears anywhere in a forwarded or quoted chain within the email, the email is responsive. Long digest emails that touch an issue in one segment are still responsive to that issue.
  • Forwards & replies. "FW:" / "RE:" with responsive content count. A bare forward whose only responsive content is the quoted material is still responsive (the document counts, not its novelty).
  • Mere mention vs. concerning. A passing, incidental mention with no substantive connection (e.g., a person's name in a signature block, an unrelated calendar line) is non-responsive. Substantive discussion, requests, scheduling about the issue, transmittals of issue material, and status updates are responsive.
  • Administrative / logistics. Scheduling a meeting, transmitting a document, or routing a constituent letter about a responsive issue is responsive (grade 1), even if short. It becomes key (2) only if substantively important.
  • Constituent mail & form letters. Incoming citizen mail and the office's replies about an issue are responsive. Identical form-letter responses are each responsive (do not de-dup for grading).
  • Spam / newsletters / press clippings. Responsive only if the content substantively concerns the issue; generic distribution lists are otherwise 0.
  • Ambiguous / unreadable. If redaction or truncation leaves intent unclear but issue-relevant signals are present, code 1.
  • Geographic qualifier. Most issues are scoped to Florida (see each definition). Climate change (409) is explicitly not Florida-restricted.
  • Negation still counts. A document arguing against a position, or reporting that something did not happen, still concerns the issue → responsive.

3. Two-tier relevance: what makes a document "Key" (grade 2)

Grade 2 is reserved for documents materially significant to the issue. Typical signals:

  • Originates or transmits a substantive report, analysis, legislation, draft policy, or legal memo on the issue.
  • Records a decision, directive, commitment, or the Governor's own position.
  • Contains non-public, deliberative, or strategy discussion among principals.
  • Is the primary source document others merely reference.

Typical grade-1-not-2 documents: scheduling, routine acknowledgments, brief FYIs, constituent form replies, tangential references, news forwards without analysis.


4. The nine responsive issues

For each: Definition (controlling statement) · Include · Exclude · Decision signals · Key (grade 2).

401 — Summer Olympics

  • Definition. All documents concerning a bid to host the Summer Olympic Games in Florida.
  • Include. Florida Olympic bid committees; proposals, feasibility/economic studies, venue or infrastructure planning tied to a bid; correspondence with the USOC/IOC about a Florida bid; legislative or budget support for a bid; Tampa/Miami/Orlando/Jacksonville bid efforts.
  • Exclude. Athletes/teams competing in the Olympics with no bid connection; the Olympics as a generic event reference; non-Florida bids unless they bear on Florida's.
  • Signals. "Olympic bid," "host city," "USOC," "Games," "bid committee," candidate-city names + Olympics.
  • Key. Bid strategy, government commitments/funding decisions, formal proposals or feasibility reports.

402 — Space

  • Definition. All documents concerning the space industry, the space program, space travel (manned or unmanned, public or private), and the study or exploration of space, in Florida.
  • Include. NASA / Kennedy Space Center operations and funding; Florida aerospace industry and jobs; commercial spaceflight and spaceports; launches; space research/education in Florida; Space Florida / Spaceport authority matters.
  • Exclude. "Space" in the sense of office/real-estate space; metaphorical uses; non-space defense aviation unless space-related.
  • Signals. "NASA," "Kennedy Space Center," "Cape Canaveral," "launch," "shuttle," "aerospace," "spaceport," "satellite."
  • Key. Industry-retention or funding decisions, major program correspondence, post-incident (e.g., shuttle) policy discussion, spaceport authorizations.

403 — Bottled Water

  • Definition. All documents concerning the extraction of water in Florida for bottling by commercial enterprises.
  • Include. Permits/applications for water withdrawal for bottling; Water Management District actions on bottling; disputes over springs/aquifer extraction by bottlers (e.g., Nestlé/large bottlers); legislation/regulation of commercial water bottling; citizen opposition to bottling permits.
  • Exclude. Drinking-water safety, utilities, or general water policy not tied to commercial bottling; bottled water as a catering/supply line item.
  • Signals. "bottling," "water withdrawal/extraction permit," "springs," "Water Management District," bottler company names.
  • Key. Permit decisions, regulatory/legal analysis, Governor's-office position on a bottling controversy.

404 — Eminent Domain

  • Definition. All documents concerning the legality or morality of expropriating land in Florida for commercial development.
  • Include. Eminent-domain / condemnation for private or commercial development; reaction to Kelo v. New London; Florida legislative responses limiting eminent domain; property-rights debate over takings for development.
  • Exclude. Routine government land acquisition for clearly public use (roads, schools, conservation) absent the commercial-development/property-rights debate; ordinary real-estate transactions.
  • Signals. "eminent domain," "condemnation," "takings," "Kelo," "property rights," "blight," "redevelopment."
  • Key. Draft legislation, legal/constitutional analysis, the Governor's stated position on takings for development.

405 — Newt Gingrich

  • Definition. All documents concerning House Speaker Newt Gingrich or any entities or personnel associated with Newt Gingrich.
  • Include. Gingrich directly; his organizations, staff, and close associates; correspondence to/from or about him; invitations, endorsements, policy collaboration involving Gingrich or his entities.
  • Exclude. Unrelated individuals who merely share the surname; generic GOP material with no Gingrich nexus.
  • Signals. "Gingrich," "Newt," his organizations/PACs and named associates.
  • Key. Direct correspondence, substantive political/policy coordination, decisions involving him.

406 — Felon Disenfranchisement

  • Definition. All documents concerning the right of felons to vote in Florida, including (but not limited to) voter purges and reinstatement of voter rights. Individual clemency cases are NOT responsive.
  • Include. Restoration-of-civil-rights policy; clemency-board rules on voting rights as policy; felon voter-roll purges; litigation/legislation on felon voting; statistics and advocacy on felon enfranchisement.
  • Exclude. Individual clemency petitions / individual rights-restoration cases (explicitly carved out); general clemency unrelated to voting; ordinary criminal-justice matters.
  • Signals. "restoration of civil rights," "felon voting," "voter purge," "disenfranchisement," "clemency rules" (policy, not a named applicant).
  • Key. Policy decisions, rule changes, purge directives, litigation strategy.

407 — Faith-Based Initiatives

  • Definition. All documents concerning grants or other initiatives in Florida to offload social services to so-called faith-based agencies. Services include but are not limited to education, prisons, and emergency relief.
  • Include. Faith-based grants/contracts for social services; faith-based prison or education programs; the state office/initiative for faith-based and community programs; church-state debate over such funding.
  • Exclude. Religious references with no service-delivery/funding nexus; routine interfaith greetings/proclamations; secular nonprofit contracting.
  • Signals. "faith-based," "faith and community," "InnerChange," "ministry" + grant/contract/program, "charitable choice."
  • Key. Grant/contract decisions, program authorizations, policy or legal (Establishment Clause) analysis.

408 — Invasive Species

  • Definition. All documents concerning the problem of invasive species in Florida — non-native plants or animals that threaten the Florida ecosystem.
  • Include. Specific invasives (pythons, lionfish, melaleuca, hydrilla, fire ants, citrus canker as invasive pest, etc.); eradication/control programs; agricultural or Everglades impacts; legislation/funding on invasives.
  • Exclude. Native pests/disease unrelated to non-native invasion; general environmental or agriculture matters without an invasive-species nexus.
  • Signals. "invasive," "non-native," "exotic species," named invasives, "eradication," "infestation."
  • Key. Control-program decisions, scientific assessments, funding/policy responses.

409 — Climate Change

  • Definition. All documents concerning climate change, global warming, or carbon emissions — whether in Florida or otherwise (NOT Florida-restricted).
  • Include. Climate science/policy; greenhouse-gas / carbon-emission regulation; sea-level-rise and climate-impact discussion; Kyoto/federal climate policy; renewable-energy or emissions initiatives framed around climate.
  • Exclude. Weather, hurricanes, or general environmental topics with no climate/emissions framing; "climate" used figuratively (e.g., "business climate," "political climate").
  • Signals. "climate change," "global warming," "greenhouse gas," "carbon emissions," "CO2," "Kyoto," "sea level rise."
  • Key. Policy positions, emissions-initiative decisions, substantive analyses.

5. Notes for AI reviewers

  • Apply exactly the §2 general rules and §4 definitions — including the explicit carve-outs (406 individual clemency; 409 not Florida-bound; 402 not office "space"). These carve-outs are the most common error sources.
  • Emit a calibrated confidence with every issue call (0.0–1.0). Confidence is scored for calibration; do not emit 1.0 unless the call is certain.
  • You have no information about how many documents in this review are responsive. Do not assume a base rate; judge each document on its content against the definitions alone.

Back to Data and reproduction · Download the 3,000 document numbers (CSV)