LO
Data engineer — tasks, one by one
The unit of analysis is the task, not the job title. Each one below carries its direction, whether the judgement rests on evidence or on platform inference, the reasoning, and what it does not establish.
Every task on this page#
Building the pipeline
Automating✓ Evidence-backedExtracting from a source, reshaping it, loading it somewhere queryable, and scheduling the whole thing.
Connectors to common sources are now bought rather than written, and the transformation code is well-specified enough that generation handles it. Two decades of this work being repetitive across companies is exactly the condition that makes a task cheap.
The pipeline running is not the same as the numbers being right. Building it was never where the time went in a mature data team; keeping it true was.
When the data is wrong and nothing errored
Still human-led✓ Evidence-backedA field changed meaning upstream, a job ran twice, a timezone shifted — and every dashboard is still green.
Silent wrongness is the defining failure of this work, and it is not a detection problem that better tooling solves — it requires knowing what the number is supposed to mean to the business, which lives in people rather than in the schema. Tests catch what you thought to test.
A judgement about the nature of the failure rather than a measurement of how often it happens. Silent data errors are not quantified anywhere, partly because by definition they are found late or not at all.
Owning what a number means
Still human-led✓ Evidence-backedDefining active user, revenue, churn — and holding that definition when two teams want it to mean different things.
Text-to-SQL tools make querying cheap and thereby make definitions more load-bearing, not less: the moment anyone can ask, the answer depends entirely on which definition the model picked up. Evidence on the analyst page points the same way — on a benchmark built from real enterprise warehouses a plain model scored zero, while scoring 80 to 90% on public benchmarks, and the gap is schema and semantics rather than SQL.
That benchmark is about querying rather than about who owns definitions, and its authors sell an alternative system, which the record on the analyst page states. It establishes that enterprise schemas defeat current models; it does not establish that anyone is being hired to maintain semantics.
What it costs to keep asking
Being augmented≈ Platform inferencePartitioning, storage tiers, the query somebody scheduled hourly that scans everything, and the bill at the end of the month.
Warehouses now surface the expensive query themselves and suggest the fix, which is real help. What is left is the decision about whether the answer is worth the money, and that requires knowing who uses the number and for what.
Says nothing about whether costs are rising or falling overall, and cheap querying tends to increase the number of questions asked, which can move the bill in either direction.
What must be kept, deleted, or never collected
Still human-led≈ Platform inferenceRetention windows, deletion requests, where personal data is allowed to sit, and being able to show it.
This is a legal obligation implemented in pipelines, and legal obligations attach to a person rather than to a system. Tooling can enforce a rule once someone has decided it; deciding it, and being able to demonstrate compliance afterwards, is the part that stays.
Requirements differ sharply by market and by data type, and what would settle this task is an enforcement action naming a data team's retention or deletion practice rather than the company as a whole — a regulator saying who was supposed to delete what, and by when.
Feeding the systems that answer in sentences
New task✓ Evidence-backedGetting the right documents, the right freshness and the right access rules into whatever the AI feature reads from.
Retrieval quality is now a data problem wearing an AI label, and it lands here because it is about pipelines, permissions and freshness rather than about models. It did not exist as a job before generative features shipped into products.
New work appearing is not new headcount. In most companies this is being absorbed by whoever already owns the warehouse, and nothing here says otherwise.