LO
Data engineer
Moves data from where it happens to where it is asked about, and is the person who finds out first when the two stop matching.
This is not a probability of losing your job. It combines how much of the role's task load is exposed to automation with how far adoption has actually gone — useful for comparing occupations on one consistent basis, and for nothing else.
Written for engineers who build and run pipelines, warehouses and the models that sit on them. The data-analyst page covers the people who query the result; machine-learning engineering is a different job this site does not yet cover. This page closes an IOU: the analyst page's scope has always said data engineering is different.
The evidence base holds verified records for other occupations, but not one for this one yet. Until it does, the analysis below is reasoning about task structure and known technical capability — for this job in particular it is not backed by traceable sources, and we would rather say so than cite things we have not verified. An empty section here is a gap in our coverage, not a finding about the work.
What is actually changing#
The unit of analysis is the task, not the job title. A role is not replaced — its task mix shifts.
Building the pipeline
Automating≈ Platform inferenceExtracting from a source, reshaping it, loading it somewhere queryable, and scheduling the whole thing.
Connectors to common sources are now bought rather than written, and the transformation code is well-specified enough that generation handles it. Two decades of this work being repetitive across companies is exactly the condition that makes a task cheap.
The pipeline running is not the same as the numbers being right. Building it was never where the time went in a mature data team; keeping it true was.
When the data is wrong and nothing errored
Still human-led≈ Platform inferenceA field changed meaning upstream, a job ran twice, a timezone shifted — and every dashboard is still green.
Silent wrongness is the defining failure of this work, and it is not a detection problem that better tooling solves — it requires knowing what the number is supposed to mean to the business, which lives in people rather than in the schema. Tests catch what you thought to test.
A judgement about the nature of the failure rather than a measurement of how often it happens. We hold no record of teams quantifying silent data errors, partly because by definition they are found late or not at all.
Owning what a number means
Still human-led≈ Platform inferenceDefining active user, revenue, churn — and holding that definition when two teams want it to mean different things.
Text-to-SQL tools make querying cheap and thereby make definitions more load-bearing, not less: the moment anyone can ask, the answer depends entirely on which definition the model picked up. Evidence on the analyst page points the same way — on a benchmark built from real enterprise warehouses a plain model scored zero, while scoring 80 to 90% on public benchmarks, and the gap is schema and semantics rather than SQL.
That benchmark is about querying rather than about who owns definitions, and its authors sell an alternative system, which the record on the analyst page states. It establishes that enterprise schemas defeat current models; it does not establish that anyone is being hired to maintain semantics.
What it costs to keep asking
Being augmented≈ Platform inferencePartitioning, storage tiers, the query somebody scheduled hourly that scans everything, and the bill at the end of the month.
Warehouses now surface the expensive query themselves and suggest the fix, which is real help. What is left is the decision about whether the answer is worth the money, and that requires knowing who uses the number and for what.
Says nothing about whether costs are rising or falling overall, and cheap querying tends to increase the number of questions asked, which can move the bill in either direction.
What must be kept, deleted, or never collected
Still human-led≈ Platform inferenceRetention windows, deletion requests, where personal data is allowed to sit, and being able to show it.
This is a legal obligation implemented in pipelines, and legal obligations attach to a person rather than to a system. Tooling can enforce a rule once someone has decided it; deciding it, and being able to demonstrate compliance afterwards, is the part that stays.
Requirements differ sharply by market and by data type, and this page holds no verified record about enforcement against data teams specifically.
Feeding the systems that answer in sentences
New task≈ Platform inferenceGetting the right documents, the right freshness and the right access rules into whatever the AI feature reads from.
Retrieval quality is now a data problem wearing an AI label, and it lands here because it is about pipelines, permissions and freshness rather than about models. It did not exist as a job before generative features shipped into products.
New work appearing is not new headcount. In most companies this is being absorbed by whoever already owns the warehouse, and nothing here says otherwise.
Which technologies matter here#
Four separate signals. They are deliberately not added together — a job exposed to two technologies is not twice as exposed.
How it got here#
The index is not a static number. This is where it would have sat at each capability checkpoint since ChatGPT — reconstructed, and labelled as such.
The highest starting point of the three engineering layers, because this work was already being commoditised before generative tools existed — bought connectors and managed warehouses had been eating the building half for years. The middle rise is transformation code becoming cheap to draft and natural-language querying arriving. It flattens earliest and stops rising at all after 2025, and the reason is on the page: cheap querying makes definitions more load-bearing, not less. The one benchmark we hold that was built from real corporate warehouses scores a plain model at zero where public benchmarks read 80 to 90 — the gap is meaning, and meaning lives in people.
A flat line is not a forecast of safety. It says which tasks automation has reached so far — the occupations that moved least here are the ones where the constraint is physical or regulatory, and both of those can change.
Recent changes#
No verified events recorded yet.
This section will fill from the monitoring pipeline as events are collected, de-duplicated, graded and linked to the tasks above. An empty list here means we have not verified anything — it does not mean nothing is happening.
"We found no news" is not the same as "you are safe."
What this means for you#
Pipeline building — the thing tutorials teach and the thing you would be hired to do — is the exposed half. What holds is knowing what a number is supposed to mean, and that is learned by being close to the people who use it. A data job far from the business is the riskier version of this role.
Cheap querying raises the value of whoever owns definitions and lowers the value of whoever only moves data. If your week is mostly connectors and scheduling, that is the exposed configuration; if it is mostly arguments about what counts as a customer, that is the durable one.
Your options#
Four directions, each with its real constraints and one thing you can test this week. Continuing as you are is a legitimate choice — it just has to be a chosen one.
Own the semantics, not the pipes
When anyone can ask a question in words, the answer depends on whose definition the tool picked up. Being the person who decides that is the durable position.
It is political work as much as technical — definitions are contested because different teams are measured on them.
Take one metric on a company dashboard and ask two teams what it means. If the answers differ, you have found the work.
Take the retrieval layer
Every AI feature in a company eventually becomes a question about which documents, how fresh, and who is allowed to see them — which is this job with a new name.
It is currently unglamorous and often unowned, and you may have to claim it before anyone budgets for it.
Find one AI feature in your company and trace what it actually reads from. Check whether its permissions match the source system's.
Move toward the decision
Analytics engineering and product analytics sit between the warehouse and the people who act on it, and the reasoning transfers directly.
Requires being willing to be wrong in public about what a number means, which is a different exposure from being wrong in a pipeline.
Pick one number you produce and find out what decision was actually made because of it last quarter. Sometimes the answer is none.
Common questions#
Ask it per task, because they are not moving together. Building and scheduling pipelines is the exposed part and has been getting cheaper since long before generative tools. Knowing that a number quietly stopped meaning what it used to mean has not moved at all, because the knowledge is about the business rather than the data. A signal you can check yourself: when a number was wrong last quarter, who noticed, and how.
The evidence on this site points the other way, and specifically. On a benchmark built from real enterprise warehouse query logs, a plain model scored zero and agent approaches reached around 10%, while the same class of model scores 80 to 90% on public benchmarks. The gap is not SQL — it is schemas with thousands of tables and meanings that live outside them. Cheap querying raises the value of whoever maintains that meaning.
It depends which half you would be doing, and that is worth settling before you move. Roles that are mostly connectors and scheduling sit in the exposed half and have been commoditising for years. Roles close to the business, where you are expected to argue about definitions, are the durable half. The job title is the same in both cases, so ask in the interview what the last three arguments were about.
How the business makes money, in enough detail to notice when a number stops being consistent with it. Tools change every few years and every one of them is learnable in weeks; the ability to look at a dashboard and say that cannot be right is what distinguishes the people who stay. It also happens to be the thing a model reading your schema cannot do.
Method and sources#
- Assessment date
- 2026-09-12
- Basis of the task judgements
- 0 evidence-backed · 6 platform inference · 0 not enough evidence
- Verified events
- 0