# On Beaver, a text-to-SQL benchmark built from real corporate warehouse query logs, a plain LLM scored zero and an agentic setup reached about 10%, against 80-90%+ on the public benchmarks

Page: https://flyvolo.ai/en/changes/ev-20260720-data-analyst-2
Date: 2026-07-20 · Stage: Constraint
Can this record move a task judgement? yes
Occupation: https://flyvolo.ai/en/careers/data-analyst

## Scope

Written by the authors of the benchmark, who also have a competing system (Rubicon) to promote — so the interested party here is arguing that the easy benchmarks are wrong. The underlying claim is checkable: Beaver is built from real query logs at MIT's 1,400-table Oracle warehouse and three others, and its leaderboard is public. The four reasons given are structural rather than about model quality — public benchmark data is in the training corpus, real schemas rot into six different columns named 'salary', warehouses carry local idiom, and real queries join two or three tables rather than one.

## Source

- BLOG@CACM (Stonebraker & Chen, MIT) — https://cacm.acm.org/blogcacm/if-you-think-you-can-do-real-world-text-to-sql (primary source)

## What this means

The gap between 90% on a public benchmark and 10% on a real warehouse is not a model problem that the next release fixes; it is the shape of actual corporate data. The reason your job survived the last three text-to-SQL products is written down here: the schema rot, the local idiom, the fact that six columns are called salary and only you know which is which. That knowledge is the job — more of it than the SQL is.

## What it does not show yet

One benchmark, four warehouses, and the authors sell an alternative. Nothing here says a company kept its analysts or stopped buying these tools — plenty of software is bought on the 90% number. The result also may not hold for every warehouse: the reasons given are structural, so a clean schema with simple queries and little local idiom is a different case.

## How to verify it yourself

Run the test on your own warehouse instead of trusting either number. Take ten real questions from last quarter's requests, give a model your schema and nothing else, and check the SQL against what you actually shipped. Count how many it got right without being told which tables to use. That number is your job security, measured, and it is also the list of things worth documenting if it comes out high.


More: https://flyvolo.ai/llms.txt