# On 632 enterprise data workflows drawn from real warehouses, a code agent solved 21.3% — against 91.2% on the earlier academic benchmark Spider 1.0

Page: https://flyvolo.ai/en/changes/ev-20241112-data-engineer-1
Date: 2024-11-12 · Stage: Constraint
Can this record move a task judgement? yes
Occupation: https://flyvolo.ai/en/careers/data-engineer

## Scope

632 problems built from real data applications on BigQuery and Snowflake, with databases often exceeding 1,000 columns. What makes them hard is stated by the authors and is exactly this occupation's daily environment: the answer requires searching database metadata and dialect documentation, reading the project's own codebase, holding very long context, and emitting several queries in different dialects that frequently run past 100 lines. The 21.3% is one agent framework on o1-preview at the end of 2024 and will be out of date; the durable finding is the gap to 91.2% on Spider 1.0 and 73.0% on BIRD, which is a measurement of how much of the difficulty lives in the warehouse rather than in the SQL. It says nothing about employment, and nothing about pipelines that are built rather than queried - it tests producing the query, not operating the system that keeps producing it.

## Source

- Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows (arXiv:2411.07763) — https://arxiv.org/abs/2411.07763 (primary source)

## What this means

The difficulty in this job is measurable and most of it is not in the SQL. When the same class of task is posed over a real warehouse — a thousand columns, several dialects, documentation to search, a codebase to read — the success rate falls from roughly nine in ten to roughly two in ten. That gap is a number for the thing data engineers say and are rarely believed about: the query is the easy part, and knowing which table is the real one is the job.

## What it does not show yet

A benchmark score is not an employer's decision, and a low one is not safety. The number is from one agent framework on one model generation at the end of 2024 and should be assumed to have moved; what does not move as fast is why it was low. It also tests producing a correct query, which is only one of this occupation's tasks — nothing here bears on building a pipeline, on what happens when the upstream schema changes, or on who is called when the numbers stop arriving.

## How to verify it yourself

Pick one number your company steers by and trace it back to the tables it is computed from. Count how many of the decisions along the way are written down anywhere. That count, not the SQL, is what a tool would have to reproduce.


More: https://flyvolo.ai/llms.txt