LO
ConstraintCognitive automation2025-07-10
A METR randomised trial of 16 experienced open-source developers on 246 real issues found they took 19% longer with early-2025 AI tools, while believing they had been 20% faster
Experienced software engineeroccupation page →
Event date / reported
2025-07-10
Evidence stage
ConstraintFailure, rollback, regulation or cost is suppressing adoption. Can lower an assessment or widen its uncertainty.
Tasks this bears on
Changing a system you did not write
Making a change inside a large, long-lived codebase where the constraints are undocumented and live in what previous decisions already committed to.
Still human-led✓ Evidence-backed
Where this applies
Experienced maintainers on large, mature repositories they know well (about 5 years each); tools were mainly Cursor Pro with Claude 3.5/3.7 Sonnet. The authors explicitly do not claim the result generalises to most developers or to greenfield or junior work.
What this means
A constraint record on the capability layer with a measured result: in a randomised trial, experienced maintainers took 19% longer with early-2025 tools on real issues in repositories they knew, while believing they were 20% faster. For the linked tasks it caps how far 'routine code is generated' may be read as 'the work got faster', and the perception gap shows developers' own sense of speed is not a reliable measure of it.
What it does not yet show
The authors say so themselves: 16 developers on mature repositories they had worked in for about five years, using tools that have since changed. It does not show junior developers get slower — they lack the repository knowledge that made the tools less useful here — and it does not measure code quality, learning, or greenfield work. A slowdown in this setting is not a slowdown in yours.
What you can check
For one week, before starting each ticket write down your time estimate with and without the assistant, then record the actual time; METR's finding was the gap between those columns, and a personal log of ten tickets shows whether you have the same gap.
Does it change the assessment?
No. The impact index is never moved by a single event. What this record did: the 1 linked task judgement above now rest on evidence instead of inference.
Source
METR — study report · verified 2026-09-12 · Claude (agent) · interpreted 2026-10-16 · Claude (COO), interpretation audit 2026-10-16
Primary source — published by the party that did this, or the authority of record. No co-signature needed.
This record is cited in