What actually gets automated, and how to tell in advance

Two companies, two layers of the stack, the same machinery

Airbnb published its account of migrating nearly 3,500 React component test files from one testing framework to another: six weeks against a hand estimate of a year and a half, with 75% of files cleared in the first four-hour bulk run. Amazon published that it moved tens of thousands of its own production Java applications to a newer Java version, estimating the manual equivalent at over 4,500 years of development work.

Neither is about writing new software. Both are about changing software that already exists and must keep working. And both worked for a reason each company states plainly: every step of the job has an automatic referee. The compiler rejects what will not build; the test suite catches most of what builds but is wrong; the linter and the type checker catch the rest. Airbnb modelled its migration as a state machine where a file only advances when the previous check passes, and then simply retried — most files finished within ten attempts.

That is the whole trick, and it is worth saying in plain words: the model was not better than a person at this. It was allowed to be wrong, cheaply, thousands of times, with nobody watching. A human engineer cannot be wrong a hundred times on one file. A loop can.

Airbnb also published the ceiling, which is the rarer thing

Three per cent of the files did not go. Each of them had been retried somewhere between fifty and a hundred times before the team stopped and finished them by hand. That number is more useful than the 97% above it, because it is a measurement of the part that automation could not reach even when retrying was nearly free.

Note also what the team says made the difference on the hard files: not better prompt wording, but choosing which related files to put in front of the model — up to fifty of them, prompts running to a hundred thousand tokens. The scarce input was knowing which parts of the codebase were relevant. That is a judgement about a specific company's code, and it was made by the people who work there.

The inverse case: take the referee away and the direction reverses

A randomised controlled trial run by METR gave sixteen experienced open-source maintainers 246 real issues in codebases they knew well. With 2025-era tools they were 19% slower. They believed they had been 20% faster. The same technology, applied to work where the correct answer is not mechanically checkable and the developer already holds the context, moved the number the other way — and moved the developers' perception in the opposite direction from the measurement.

That last part is why self-report is not evidence on this site. The people in the trial were experienced, motivated, and wrong about their own speed by roughly forty percentage points.

Four benchmarks say the same thing from the other side

Spider 2.0 poses enterprise data questions over real warehouses — a thousand columns, several SQL dialects, documentation and a project codebase to search. A code agent solved 21.3% of them, against 91.2% on the academic version of the same task. Beaver, built from real corporate warehouse query logs, scores a plain language model at zero where public benchmarks read 80 to 90. On the Finance Agent Benchmark's 537 expert-authored questions over recent filings, the best models fall far short of an analyst.

OpenAI's GDPval, which collects real deliverables from 44 occupations and has industry experts grade model output blind against them, reports the gradient directly: win rates are highest on tasks taking nought to two hours and decline steadily as the task gets longer. The longer a task runs, the more of it consists of decisions that nothing can check until the end.

None of these is a statement about how smart a model is. They are statements about where the referee stops working. When the correct answer depends on which table in this company is the real one, or which of two definitions of a metric the business actually uses, or what the client actually meant, nothing but a person can say so.

What happens when the work ships without a referee

US federal courts have sanctioned lawyers for filings built on citations to cases that do not exist — a $5,000 fine in one instance, revoked admission in another. A syndicated summer reading list ran in the Chicago Sun-Times and the Philadelphia Inquirer containing books that were never written. Crunchyroll's German subtitles for an anime premiere contained the line 'ChatGPT said…'. Florida's Supreme Court has since amended its rules so that signing a filing is itself a representation about the legal authority cited in it.

These are not funny failures, they are the same structure as the successes. Where the checking step was mechanical, the machine did it. Where the checking step was a person, and the person was removed to save the checking step, the error reached the outside world — and then a court, an editor or a regulator became the referee instead, after the fact and at much higher cost.

In physical work, the bounded, repeated job goes first

The same shape shows up outside software. Mowing is the one grounds task where machines are now employed at scale: Singapore's parks board says landscape companies saved 80% of man-hours after switching to autonomous robotic mowers, and Nashville's council contracted robotic mowers for its parks. A lawn has edges a machine can be told and a result anyone can see. Drinks made at a fixed station are the next case: robot barista coffee shops operate commercially in South Korea, where a study of 300 of their customers found atmosphere and novelty drew them. Cutting hair is the opposite: every head is different and nothing but a person says whether the cut is right, and the robots found only comb and style hair, one of them on wigs.

In the building trades, the factory moves first

In carpentry the work that leaves the site goes to factories, not to robots on site. Japan's Forestry Agency reports that 94 percent of post-and-beam houses used factory precut joints in 2024, and Germany's statistics office counts 22.2 percent of homes completed in 2024 as factory-built. Automation inside the factory is no guarantee either: Katerra, whose plant had highly automated framing lines, reported about $2.78 billion in losses before it filed for bankruptcy.

What to do with this on Monday

Take your own backlog and mark each item with what would tell you it is finished: a green check that appears without you, or a person looking at it. The first list is the one that is about to get cheaper, whatever industry you are in. Knowing how long that list is inside your own job is worth more than any published percentage, and nobody else can compute it for you.

Then look at the second list and ask a harder question: is the checking you do visible to the person who decides your budget? The recurring pattern in these records is that the review step is the first thing cut, because it produces no artefact — and its absence does not show up for two quarters.

Everything cited here

Each one opens the full record: its source and source tier, the dates, the scope it applies to, who verified it and what it does not establish.

1. Airbnb reports migrating nearly 3,500 React component test files from Enzyme to React Testing Library in six weeks, against a hand estimate of 1.5 years — The Airbnb Tech Blog (Airbnb's own engineering blog)
2. Amazon says it moved tens of thousands of its own production Java applications from Java 8 or 11 to Java 17 with an agent, and estimates the manual equivalent at over 4,500 years of development work — AWS DevOps & Developer Productivity Blog (Amazon's own)
3. A METR randomised trial of 16 experienced open-source developers on 246 real issues found they took 19% longer with early-2025 AI tools, while believing they had been 20% faster — METR — study report
4. On 632 enterprise data workflows drawn from real warehouses, a code agent solved 21.3% — against 91.2% on the earlier academic benchmark Spider 1.0 — Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows (arXiv:2411.07763)
5. On Beaver, a text-to-SQL benchmark built from real corporate warehouse query logs, a plain LLM scored zero and an agentic setup reached about 10%, against 80-90%+ on the public benchmarks — BLOG@CACM (Stonebraker & Chen, MIT)
6. On the Finance Agent Benchmark — 537 expert-authored questions over recent SEC filings — the best model, OpenAI o3, reached 46.8% accuracy at $3.79 per query — arXiv 2508.00828 (Bigeard, Nashold, Krishnan, Wu)
7. OpenAI's GDPval benchmark collected real purchasing-agent deliverables and had industry experts grade model output against them blind — GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks (OpenAI, arXiv:2510.04374)
8. A US federal court fined two attorneys and their firm $5,000 for a brief citing six non-existent cases generated by ChatGPT, noting that using a reliable AI tool is not itself improper — Mata v. Avianca, Inc. (S.D.N.Y., 22 June 2023) — opinion and order on sanctions, via CourtListener
9. A Wyoming federal court fined three attorneys and revoked one's pro hac vice admission after a motion they filed cited nine cases, eight of which did not exist — US District Court, D. Wyo. — Wadsworth v. Walmart, 2:23-cv-118-KHR, ECF No. 181 (order of Judge Kelly H. Rankin)
10. Florida's Supreme Court amends Rule 2.515(d)(2): signing a filing represents that the legal authorities cited exist and are accurate, with sanctions available, effective June 15, 2026 — Supreme Court of Florida, No. SC2026-0673 — In re: Amendments to Florida Rule of General Practice and Judicial Administration 2.515 (2026-05-28)
11. The Chicago Sun-Times and Philadelphia Inquirer printed a syndicated summer reading list with books that do not exist; the paper said the freelancer used an AI tool and the section was not reviewed — NBC News
12. Crunchyroll's German subtitles for an anime premiere contained the line 'ChatGPT said…'; the company said a third-party vendor had used AI-generated subtitles in violation of its agreement — Engadget
13. Landscape companies saw 80% savings in man-hours after switching to autonomous robotic lawn mowers, Singapore's national parks board said — National Parks Board (NParks), Singapore — NParks rolls out new model of greenery management for parks and streetscape that leverages suite of digital tools developed with landscape industry (29 Mar 2023)
14. Leased robotic mowers for its parks department were approved by the Nashville council in a five-year contract with an estimated value of $2,000,000 — Metropolitan Government of Nashville and Davidson County — Resolution RS2025-1015, approving a contract with My Goat, Inc. for robotic mowers for the Department of Parks and Recreation (passed 2025-02-18; Legistar record)
15. Customers of robot barista coffee shops in South Korea were drawn by atmosphere and novelty, a survey of 300 visitors to four operating chains found — Kim, Yoo, Jeon (Gachon University and others) — The role of experiential value, novelty, and satisfaction in robot barista coffee shop in South Korea: COVID-19 crisis and beyond, Service Business (published 31 October 2021)
16. A robot hair-styling system trained in simulation carried its skill over to wigs, researchers reported, calling hair care challenging for robots because of its fine structure — Zhao, Yoo, Chaudhury, Nam, Francis, Ichnowski — DYMO-Hair: Generalizable Volumetric Dynamics Modeling for Robot Hair Manipulation, arXiv 2510.06199 (7 Oct 2025)
17. Factory precutting reached 94 percent of post-and-beam timber houses in 2024, Japan's Forestry Agency reported in its white paper — 林野庁 — 令和7年度 森林・林業白書(令和8年6月2日公表), 第III章
18. Factory-built houses made up 22.2 percent of residential buildings completed in Germany in 2024, the federal statistics office reported — Statistisches Bundesamt (Destatis) — Pressemitteilung Nr. N039 vom 28. Juli 2025
19. A factory with highly automated lines for wood-framed walls and trusses belonged to a builder that lost about $2.78 billion and filed for bankruptcy, Katerra declared in court — U.S. Bankruptcy Court, Southern District of Texas, Case 21-31861 (Katerra Inc.), Document 37 — declaration in support of Chapter 11 petitions, filed 06/07/21 (via CourtListener RECAP)
20. Getting people to actually use it — task on AI implementation lead
21. Making the numbers mean something — task on Business systems owner