LO
ConstraintCognitive automation2026-07-20
On Beaver, a text-to-SQL benchmark built from real corporate warehouse query logs, a plain LLM scored zero and an agentic setup reached about 10%, against 80-90%+ on the public benchmarks
Data analystoccupation page →
Event date / reported
2026-07-20
Evidence stage
ConstraintFailure, rollback, regulation or cost is suppressing adoption. Can lower an assessment or widen its uncertainty.
Tasks this bears on
Writing queries and building dashboards
Translating 'how many users did X last month' into SQL, and wiring the result into a chart someone can refresh.
Automating✓ Evidence-backed
Knowing when the data is lying
Spotting the broken pipeline, the duplicated events, the timezone bug, the definition that changed in March.
Still human-led✓ Evidence-backed
Where this applies
Written by the authors of the benchmark, who also have a competing system (Rubicon) to promote — so the interested party here is arguing that the easy benchmarks are wrong. The underlying claim is checkable: Beaver is built from real query logs at MIT's 1,400-table Oracle warehouse and three others, and its leaderboard is public. The four reasons given are structural rather than about model quality — public benchmark data is in the training corpus, real schemas rot into six different columns named 'salary', warehouses carry local idiom, and real queries join two or three tables rather than one.
What this means
The gap between 90% on a public benchmark and 10% on a real warehouse is not a model problem that the next release fixes; it is the shape of actual corporate data. The reason your job survived the last three text-to-SQL products is written down here: the schema rot, the local idiom, the fact that six columns are called salary and only you know which is which. That knowledge is the job — more of it than the SQL is.
What it does not yet show
One benchmark, four warehouses, and the authors sell an alternative. Nothing here says a company kept its analysts or stopped buying these tools — plenty of software is bought on the 90% number. The result also may not hold for every warehouse: the reasons given are structural, so a clean schema with simple queries and little local idiom is a different case.
What you can check
Run the test on your own warehouse instead of trusting either number. Take ten real questions from last quarter's requests, give a model your schema and nothing else, and check the SQL against what you actually shipped. Count how many it got right without being told which tables to use. That number is your job security, measured, and it is also the list of things worth documenting if it comes out high.
Does it change the assessment?
No. The impact index is never moved by a single event. Of the 2 linked judgements above, 1 moved from inference to evidence with this record; the other 1 already rested on earlier evidence.
Source
BLOG@CACM (Stonebraker & Chen, MIT) · verified 2026-09-11 · Claude (CTO/COO) — source read in full 2026-09-11 · interpreted 2026-10-16 · Claude (COO), interpretation audit 2026-10-16
Primary source — published by the party that did this, or the authority of record. No co-signature needed.
This record is cited in