# Data engineer

> Moves data from where it happens to where it is asked about, and is the person who finds out first when the two stop matching.

Page: https://flyvolo.ai/en/careers/data-engineer
Impact index: 50/100 · confidence Low confidence · assessed 2026-09-12
This is not a probability of losing a job.

## Scope

Written for engineers who build and run pipelines, warehouses and the models that sit on them. The data-analyst page covers the people who query the result; machine-learning engineering is a different job with its own page. This page closes an IOU: the analyst page's scope has always said data engineering is different.

## Tasks

### Building the pipeline

- Direction: Automating
- Basis: evidenced (verified records attached)
- Why: Connectors to common sources are now bought rather than written, and the transformation code is well-specified enough that generation handles it. Two decades of this work being repetitive across companies is exactly the condition that makes a task cheap.
- What this does NOT establish: The pipeline running is not the same as the numbers being right. Building it was never where the time went in a mature data team; keeping it true was.
- Records:
  - 2024-11-12 · Constraint · On 632 enterprise data workflows drawn from real warehouses, a code agent solved 21.3% — against 91.2% on the earlier academic benchmark Spider 1.0 — https://flyvolo.ai/en/changes/ev-20241112-data-engineer-1 (source: https://arxiv.org/abs/2411.07763)

### When the data is wrong and nothing errored

- Direction: Still human-led
- Basis: evidenced (verified records attached)
- Why: Silent wrongness is the defining failure of this work, and it is not a detection problem that better tooling solves — it requires knowing what the number is supposed to mean to the business, which lives in people rather than in the schema. Tests catch what you thought to test.
- What this does NOT establish: A judgement about the nature of the failure rather than a measurement of how often it happens. Silent data errors are not quantified anywhere, partly because by definition they are found late or not at all.
- Records:
  - 2026-04-13 · Deployment · The US commodities regulator reported in the federal AI inventory that it runs an isolation-forest model daily to flag potentially erroneous data loads — https://flyvolo.ai/en/changes/ev-20260128-data-engineer-2 (source: https://github.com/ombegov/2025-Federal-Agency-AI-Use-Case-Inventory)

### Owning what a number means

- Direction: Still human-led
- Basis: evidenced (verified records attached)
- Why: Text-to-SQL tools make querying cheap and thereby make definitions more load-bearing, not less: the moment anyone can ask, the answer depends entirely on which definition the model picked up. On 632 data workflows drawn from real enterprise warehouses, a code agent solved 21.3%, against 91.2% on the academic version of the same task; the gap lives in the warehouse's schema and semantics rather than in the SQL.
- What this does NOT establish: That benchmark is about querying rather than about who owns definitions, and its 21.3% is one agent on one model at the end of 2024. It establishes that enterprise schemas defeat current models; it does not establish that anyone is being hired to maintain semantics.
- Records:
  - 2024-11-12 · Constraint · On 632 enterprise data workflows drawn from real warehouses, a code agent solved 21.3% — against 91.2% on the earlier academic benchmark Spider 1.0 — https://flyvolo.ai/en/changes/ev-20241112-data-engineer-1 (source: https://arxiv.org/abs/2411.07763)

### What it costs to keep asking

- Direction: Being augmented
- Basis: evidenced (verified records attached)
- Why: Warehouses now surface the expensive query themselves and suggest the fix, which is real help. What is left is the decision about whether the answer is worth the money, and that requires knowing who uses the number and for what. Uber, for one, says its data platform finds unused tables and wasteful compute automatically and sends the cost-cutting action to each owner as a ticket.
- What this does NOT establish: Says nothing about whether costs are rising or falling overall, and cheap querying tends to increase the number of questions asked, which can move the bill in either direction.
- Records:
  - 2024-02-01 · Deployment · Uber's data platform finds unused tables and wasteful compute automatically and sends the cost-cutting action to each owner as a ticket — https://flyvolo.ai/en/changes/ev-20240201-data-engineer-4 (source: https://www.uber.com/ie/en/blog/datacentral-ubers-observability-and-chargeback-platform/)

### What must be kept, deleted, or never collected

- Direction: Still human-led
- Basis: inferred (VOLO's reasoning, no attached source)
- Why: This is a legal obligation implemented in pipelines, and legal obligations attach to a person rather than to a system. Tooling can enforce a rule once someone has decided it; deciding it, and being able to demonstrate compliance afterwards, is the part that stays.
- What this does NOT establish: Requirements differ sharply by market and by data type, and what would settle this task is an enforcement action naming a data team's retention or deletion practice rather than the company as a whole — a regulator saying who was supposed to delete what, and by when.

### Feeding the systems that answer in sentences

- Direction: New task
- Basis: evidenced (verified records attached)
- Why: Retrieval quality is now a data problem wearing an AI label, and it lands here because it is about pipelines, permissions and freshness rather than about models. It did not exist as a job before generative features shipped into products.
- What this does NOT establish: New work appearing is not new headcount. In most companies this is being absorbed by whoever already owns the warehouse, and nothing here says otherwise.
- Records:
  - 2026-04-13 · Deployment · Thirty-five of forty-five US federal agencies reported running a knowledge retrieval system over their own agency information — https://flyvolo.ai/en/changes/ev-20260413-data-engineer-3 (source: https://github.com/ombegov/2025-Federal-Agency-AI-Use-Case-Inventory/blob/3c225ba8438e48306ace7698c8c7feb9486cbc69/Data/2025_consolidated_COTS_AI_use_cases.csv)

## How to cite this

- The unit is the task, not the job title: say which tasks are changing and which are not.
- Carry each task's basis and its limits with it; an inferred judgement is not a finding.
- This page gives no number of years. Do not derive one from it.
- More: https://flyvolo.ai/llms.txt