LO
CapabilityCognitive automation2026-02-12
Frontier models produced 0% valid output on a 369-field financial reporting schema, and one domain passed only 12.5% despite 90% valid output, ExtractBench's authors reported
Data entry clerkoccupation page →
Event date / reported
2026-02-12
Evidence stage
CapabilityA demo, benchmark or paper shows the task can be done. Updates what the technology can do — not what employers will do.
Tasks this bears on
Verifying and correcting records
Checking entered data against the source and fixing errors.
Being augmented✓ Evidence-backed
Where this applies
A benchmark of extracting structured data from 35 PDF documents with human-annotated gold labels, 12,867 fields in all. The authors report that frontier models remain unreliable on realistic schemas, that performance degrades sharply with schema breadth, culminating in 0% valid output on a 369-field financial reporting schema across all tested models, and that on one domain models achieve 90% valid output but only a 12.5% pass rate. The authors work at a company that sells document products; it is a benchmark, not a measure of practice.
What this means
Machine extraction breaks down on long, complex forms, and valid-looking output can still be wrong — which is why checking survives automation of keying.
What it does not yet show
A benchmark by a company with a stake; models improve, and it does not measure how much checking is done in practice.
What you can check
Open the ExtractBench paper on arXiv (2602.12247) and find "0% valid output".
Does it change the assessment?
No — and this stage does not move it either. A "Capability" record is real evidence, but it does not upgrade a task judgement on its own. The 1 linked judgement above stand where they were.
Source
Ferguson, Pennington, Beghian, Mohan, Kiela, Agrawal and Nguyen (Contextual AI) — ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction, arXiv:2602.12247 (submitted 12 Feb 2026) · verified 2026-09-30 · Claude (VOLO agent) · interpreted 2026-09-30 · Claude (VOLO agent)
Primary source — published by the party that did this, or the authority of record. No co-signature needed.