LO
CapabilityCognitive automation2025-01-31
Human teams reproducing published research succeeded 57 percentage points more often than AI-led teams and as often as AI-assisted ones, a randomised study of 288 researchers found
Economistoccupation page →
Event date / reported
2025-01-31
Evidence stage
CapabilityA demo, benchmark or paper shows the task can be done. Updates what the technology can do — not what employers will do.
Tasks this bears on
Data work and modelling
Building datasets, writing code, estimating models and checking that results hold up.
Being augmented✓ Evidence-backed
Research and writing
Reviewing the literature, writing papers and reports, and presenting findings.
Being augmented≈ Platform inference
Where this applies
An experiment with 288 researchers randomly assigned to 103 teams that computationally reproduced published social-science results: human-only teams, AI-assisted teams and teams that minimally guided an AI. Human teams matched AI-assisted teams' success rates and achieved 57 percentage points higher success than AI-led teams, and found significantly more major errors than both. A discussion paper using 2024 models.
What this means
Handing research checks to AI failed in a controlled test, while using AI as an assistant matched but did not beat people.
What it does not yet show
One experiment with 2024 models on reproduction tasks; it does not cover original research.
What you can check
Open IZA DP No. 17645 (January 2025) and find "57 pp higher success rates".
Does it change the assessment?
No — and this stage does not move it either. A "Capability" record is real evidence, but it does not upgrade a task judgement on its own. The 2 linked judgements above stand where they were.
Source
Brodeur A et al. — Comparing Human-Only, AI-Assisted, and AI-Led Teams on Assessing Research Reproducibility in Quantitative Social Science, IZA Discussion Paper No. 17645 (January 2025) · verified 2026-10-02 · Claude (VOLO agent) · interpreted 2026-10-02 · Claude (VOLO agent)
Primary source — published by the party that did this, or the authority of record. No co-signature needed.