What actually gets automated, and how to tell in advance

Two companies, two layers of the stack, the same machinery

Airbnb published its account of migrating nearly 3,500 React component test files from one testing framework to another: six weeks against a hand estimate of a year and a half, with 75% of files cleared in the first four-hour bulk run. Amazon published that it moved tens of thousands of its own production Java applications to a newer Java version, estimating the manual equivalent at over 4,500 years of development work.

Neither is about writing new software. Both are about changing software that already exists and must keep working. And both worked for a reason each company states plainly: every step of the job has an automatic referee. The compiler rejects what will not build; the test suite catches most of what builds but is wrong; the linter and the type checker catch the rest. Airbnb modelled its migration as a state machine where a file only advances when the previous check passes, and then simply retried — most files finished within ten attempts.

That is the whole trick, and it is worth saying in plain words: the model was not better than a person at this. It was allowed to be wrong, cheaply, thousands of times, with nobody watching. A human engineer cannot be wrong a hundred times on one file. A loop can.

Airbnb also published the ceiling, which is the rarer thing

Three per cent of the files did not go. Each of them had been retried somewhere between fifty and a hundred times before the team stopped and finished them by hand. That number is more useful than the 97% above it, because it is a measurement of the part that automation could not reach even when retrying was nearly free.

Note also what the team says made the difference on the hard files: not better prompt wording, but choosing which related files to put in front of the model — up to fifty of them, prompts running to a hundred thousand tokens. The scarce input was knowing which parts of the codebase were relevant. That is a judgement about a specific company's code, and it was made by the people who work there.

The inverse case: take the referee away and the direction reverses

A randomised controlled trial run by METR gave sixteen experienced open-source maintainers 246 real issues in codebases they knew well. With 2025-era tools they were 19% slower. They believed they had been 20% faster. The same technology, applied to work where the correct answer is not mechanically checkable and the developer already holds the context, moved the number the other way — and moved the developers' perception in the opposite direction from the measurement.

That last part is why self-report is not evidence on this site. The people in the trial were experienced, motivated, and wrong about their own speed by roughly forty percentage points.

Four benchmarks say the same thing from the other side

Spider 2.0 poses enterprise data questions over real warehouses — a thousand columns, several SQL dialects, documentation and a project codebase to search. A code agent solved 21.3% of them, against 91.2% on the academic version of the same task. Beaver, built from real corporate warehouse query logs, scores a plain language model at zero where public benchmarks read 80 to 90. On the Finance Agent Benchmark's 537 expert-authored questions over recent filings, the best models fall far short of an analyst.

OpenAI's GDPval, which collects real deliverables from 44 occupations and has industry experts grade model output blind against them, reports the gradient directly: win rates are highest on tasks taking nought to two hours and decline steadily as the task gets longer. The longer a task runs, the more of it consists of decisions that nothing can check until the end.

None of these is a statement about how smart a model is. They are statements about where the referee stops working. When the correct answer depends on which table in this company is the real one, or which of two duplicate records to keep, or what the client actually meant, nothing but a person can say so.

What happens when the work ships without a referee

US federal courts have sanctioned lawyers for filings built on citations to cases that do not exist — a $5,000 fine in one instance, revoked admission in another. A syndicated summer reading list ran in the Chicago Sun-Times and the Philadelphia Inquirer containing books that were never written. Crunchyroll's German subtitles for an anime premiere contained the line 'ChatGPT said…'. Florida's Supreme Court has since amended its rules so that signing a filing is itself a representation about the legal authority cited in it.

These are not funny failures, they are the same structure as the successes. Where the checking step was mechanical, the machine did it. Where the checking step was a person, and the person was removed to save the checking step, the error reached the outside world — and then a court, an editor or a regulator became the referee instead, after the fact and at much higher cost.

What to do with this on Monday

Take your own backlog and mark each item with what would tell you it is finished: a green check that appears without you, or a person looking at it. The first list is the one that is about to get cheaper, whatever industry you are in. Knowing how long that list is inside your own job is worth more than any published percentage, and nobody else can compute it for you.

Then look at the second list and ask a harder question: is the checking you do visible to the person who decides your budget? The recurring pattern in these records is that the review step is the first thing cut, because it produces no artefact — and its absence does not show up for two quarters.

Everything cited here

Each one opens the full record: its source and source tier, the dates, the scope it applies to, who verified it and what it does not establish.

1. Airbnb reports migrating nearly 3,500 React component test files from Enzyme to React Testing Library in six weeks, against a hand estimate of 1.5 years — The Airbnb Tech Blog (Airbnb's own engineering blog)
2. Amazon says it moved tens of thousands of its own production Java applications from Java 8 or 11 to Java 17 with an agent, and estimates the manual equivalent at over 4,500 years of development work — AWS DevOps & Developer Productivity Blog (Amazon's own)
3. A METR randomised trial of 16 experienced open-source developers on 246 real issues found they took 19% longer with early-2025 AI tools, while believing they had been 20% faster — METR — study report
4. On 632 enterprise data workflows drawn from real warehouses, a code agent solved 21.3% — against 91.2% on the academic version of the same task — Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows (arXiv:2411.07763)
5. On Beaver, a text-to-SQL benchmark built from real corporate warehouse query logs, a plain LLM scored zero and an agentic setup reached about 10%, against 80-90%+ on the public benchmarks — BLOG@CACM (Stonebraker & Chen, MIT)
6. On the Finance Agent Benchmark — 537 expert-authored questions over recent SEC filings — the best model, OpenAI o3, reached 46.8% accuracy at $3.79 per query — arXiv 2508.00828 (Bigeard, Nashold, Krishnan, Wu)
7. OpenAI's GDPval benchmark collected real purchasing-agent deliverables and had industry experts grade model output against them blind — GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks (OpenAI, arXiv:2510.04374)
8. A US federal court fined two attorneys and their firm $5,000 for a brief citing six non-existent cases generated by ChatGPT, noting that using a reliable AI tool is not itself improper — Mata v. Avianca, Inc. (S.D.N.Y., 22 June 2023) — opinion and order on sanctions, via CourtListener
9. A Wyoming federal court fined three attorneys and revoked one's pro hac vice admission after a motion they filed cited nine cases, eight of which did not exist — US District Court, D. Wyo. — Wadsworth v. Walmart, 2:23-cv-118-KHR, ECF No. 181 (order of Judge Kelly H. Rankin)
10. Florida's Supreme Court amends Rule 2.515(d)(2): signing a filing represents that the legal authorities cited exist and are accurate, with sanctions available, effective June 15, 2026 — Supreme Court of Florida, No. SC2026-0673 — In re: Amendments to Florida Rule of General Practice and Judicial Administration 2.515 (2026-05-28)
11. The Chicago Sun-Times and Philadelphia Inquirer printed a syndicated summer reading list with books that do not exist; the freelancer said he used an AI tool and the section was not reviewed — NBC News
12. Crunchyroll's German subtitles for an anime premiere contained the line 'ChatGPT said…'; the company said a third-party vendor had used AI-generated subtitles in violation of its agreement — Engadget
13. Getting people to actually use it — task on AI implementation lead
14. Making the numbers mean something — task on Business systems owner