LO
CapabilityCognitive automation2024-05-30
Commercial AI legal research tools from LexisNexis and Thomson Reuters still made things up in 17% to 33% of answers, researchers at Stanford reported
Lawyeroccupation page →
Event date / reported
2024-05-30
Evidence stage
CapabilityA demo, benchmark or paper shows the task can be done. Updates what the technology can do — not what employers will do.
Tasks this bears on
Legal research
Finding the authorities that govern a question, and the ones that cut the other way.
Being augmented✓ Evidence-backed
Where this applies
A study that tested retrieval-augmented legal research tools sold by LexisNexis (Lexis+ AI) and Thomson Reuters (Westlaw AI-Assisted Research and Ask Practical Law AI), and found each hallucinated between 17% and 33% of the time, with substantial differences between systems. It tests 2024 versions of the tools, not how lawyers or paralegals use them.
What this means
The research tools lawyers pay for still invent material often enough that every cited authority needs checking.
What it does not yet show
2024 versions of commercial tools; they may have improved, and the study does not measure use in practice.
What you can check
Open arXiv 2405.20362 and find "between 17% and 33% of the time".
Does it change the assessment?
No — and this stage does not move it either. A "Capability" record is real evidence, but it does not upgrade a task judgement on its own. The 1 linked judgement above stand where they were.
Source
Magesh et al. — Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (arXiv 2405.20362, submitted 30 May 2024) · verified 2026-09-30 · Claude (VOLO agent) · interpreted 2026-09-30 · Claude (VOLO agent)
Primary source — published by the party that did this, or the authority of record. No co-signature needed.