Data engineer

One regulator, one dataset, reported under a statutory duty. Entry CFTC-001 in the individually-reported CSV, whose whole text is two sentences: an anomaly detection model "designed to identify potentially erroneous data loads in TCR data", using "an isolation forest model with aggregated data", which "automatically runs daily". The entry carries no operational date, no vendor, no volume and no account of what the model catches or misses, so the date here is the publication of the consolidated inventory (2026-01-28) and nothing about the deployment itself is dated. It establishes that at least one data team has handed the first half of this task, noticing that a load looks wrong, to a scheduled model; it says nothing about the second half, working out what the wrong load means and what to do. The same inventory carries an entry that reads the other way: DOJ-0006, the Bureau of Alcohol, Tobacco, Firearms and Explosives, lists a data platform whose anomaly detection and data-quality features are "embedded within the tools" and adds "while we do not use these features directly". A listed capability and a used one are different things, and the inventory format lets an agency say which it has.

An employer has put it into production. Can move the baseline — weighted by scale and how similar the setting is.

The same census, thirteenth of the 20 questions: searching for agency information using a knowledge retrieval system is Y at 35 of 45. The product column names the systems themselves rather than a category — Microsoft 365 Copilot at most of them, ChatGPT Enterprise at Energy, Claude and Gemini through the GSA USAi platform, Palantir Foundry and Credal at Health and Human Services, Vertex AI and RelativityOne at Homeland Security, AWS Bedrock at Commerce and Transportation. This task is labelled emerging, meaning work that exists because of automation, and what the census establishes is its premise: at 35 of 45 agencies there is a retrieval system pointed at the documents of that agency, so there is a corpus somebody has to keep current and scope. What it does not establish is the work. The file has no column for who maintains what a system reads, no measure of effort, and nothing saying whether that job sits with a data team, a records office or the vendor. It is also one employer and an unusual one: an agency that must publish an inventory of its AI, and must answer for who may see which document, is not a fair stand-in for an employer under neither duty.

An employer has put it into production. Can move the baseline — weighted by scale and how similar the setting is.

632 problems built from real data applications on BigQuery and Snowflake, with databases often exceeding 1,000 columns. What makes them hard is stated by the authors and is exactly this occupation's daily environment: the answer requires searching database metadata and dialect documentation, reading the project's own codebase, holding very long context, and emitting several queries in different dialects that frequently run past 100 lines. The 21.3% is one agent framework on o1-preview at the end of 2024 and will be out of date; the durable finding is the gap to 91.2% on Spider 1.0 and 73.0% on BIRD, which is a measurement of how much of the difficulty lives in the warehouse rather than in the SQL. It says nothing about employment, and nothing about pipelines that are built rather than queried - it tests producing the query, not operating the system that keeps producing it.

Failure, rollback, regulation or cost is suppressing adoption. Can lower an assessment or widen its uncertainty.

One large technology employer, its own big-data platform. The engineering team writes that instead of setting hard limits and budgets on teams, Uber gives stakeholders transparency into costs and resource usage; that its cost-efficiency projects make automated, data-driven decisions to reduce costs; and that one of them checks tables' access patterns and creates tickets for owners to push for deletions and expiry when data is not frequently accessed, with similar checks raising tickets to reduce or stop unused compute. The owners act on the tickets; the post gives no figures for how many are accepted.

An employer has put it into production. Can move the baseline — weighted by scale and how similar the setting is.

Pipeline building — the thing tutorials teach and the thing you would be hired to do — is the exposed half. What holds is knowing what a number is supposed to mean, and that is learned by being close to the people who use it. A data job far from the business is the riskier version of this role.

Cheap querying raises the value of whoever owns definitions and lowers the value of whoever only moves data. If your week is mostly connectors and scheduling, that is the exposed configuration; if it is mostly arguments about what counts as a customer, that is the durable one.

Stay and strengthen

Own the semantics, not the pipes

When anyone can ask a question in words, the answer depends on whose definition the tool picked up. Being the person who decides that is the durable position.

Reshape the role

Take the retrieval layer

Every AI feature in a company eventually becomes a question about which documents, how fresh, and who is allowed to see them — which is this job with a new name.

Adjacent move

Move toward the decision

Analytics engineering and product analytics sit between the warehouse and the people who act on it, and the reasoning transfers directly.

Ask it per task, because they are not moving together. Building and scheduling pipelines is the exposed part and has been getting cheaper since long before generative tools. Knowing that a number quietly stopped meaning what it used to mean has not moved at all, because the knowledge is about the business rather than the data. A signal you can check yourself: when a number was wrong last quarter, who noticed, and how.

The evidence on this site points the other way, and specifically. On 632 data workflows drawn from real enterprise warehouses, a code agent solved 21.3%, against 91.2% on the academic version of the same task. The gap is not SQL — it is schemas that often exceed 1,000 columns and meanings that live outside them. Cheap querying raises the value of whoever maintains that meaning.

It depends which half you would be doing, and that is worth settling before you move. Roles that are mostly connectors and scheduling sit in the exposed half and have been commoditising for years. Roles close to the business, where you are expected to argue about definitions, are the durable half. The job title is the same in both cases, so ask in the interview what the last three arguments were about.

How the business makes money, in enough detail to notice when a number stops being consistent with it. Tools change every few years and every one of them is learnable in weeks; the ability to look at a dashboard and say that cannot be right is what distinguishes the people who stay. It also happens to be the thing a model reading your schema cannot do.