LO
ConstraintCognitive automation2025-02-17
A Nature Machine Intelligence study with 3,200 participants found that language models used in place of human participants, including in user testing, misportray and flatten identity groups
Product / UX designeroccupation page →
Event date / reported
2025-02-17
Evidence stage
ConstraintFailure, rollback, regulation or cost is suppressing adoption. Can lower an assessment or widen its uncertainty.
Tasks this bears on
Research and usability testing
Watching users struggle, running interviews, reading the analytics, and turning what you saw into a decision.
Being augmented✓ Evidence-backed
Where this applies
A peer-reviewed study of language models used as replacements for human participants in computational social science, user testing and annotation. Through human studies with 3,200 participants across 16 demographic identities and four models, the authors show the models misportray and flatten demographic groups, and that their own inference-time techniques reduce but do not remove these harms. It is not specific to UX work and tests models current at the time; the authors declare no competing interests.
What this means
Replacing the people in user research with simulated users has a measured cost: the simulation misrepresents the very groups a designer most needs to hear from. Watching real users stays the part of research a model cannot stand in for.
What it does not yet show
A study of models of its time, not specific to usability testing; it does not show how design teams now run research.
What you can check
Open the Nature Machine Intelligence article of 17 February 2025 and find "as replacements for human participants in computational social science, user testing, annotation tasks".
Does it change the assessment?
No. The impact index is never moved by a single event. What this record did: the 1 linked task judgement above now rest on evidence instead of inference.
Source
Wang, Morgenstern & Dickerson — "Large language models that replace human participants can harmfully misportray and flatten identity groups", Nature Machine Intelligence 7 (published 17 February 2025) · verified 2026-09-28 · Claude (VOLO agent) · interpreted 2026-09-28 · Claude (VOLO agent)
Primary source — published by the party that did this, or the authority of record. No co-signature needed.