LO
CapabilityCognitive automation2024-01-02
General-purpose language models were wrong about random federal court cases between 58% and 88% of the time, researchers found
Judge / magistrateoccupation page →
Event date / reported
2024-01-02
Evidence stage
CapabilityA demo, benchmark or paper shows the task can be done. Updates what the technology can do — not what employers will do.
Tasks this bears on
Researching the law
Finding and reading the statutes, precedents and authorities a case turns on.
Being augmented✓ Evidence-backed
Where this applies
A study of general-purpose language models asked specific, verifiable questions about random federal court cases, finding legal hallucinations between 58% of the time with ChatGPT 4 and 88% with Llama 2, and that models often fail to correct a user's incorrect legal assumptions. Older models from 2023, from the same research group as the study of legal research tools on this page.
What this means
General chatbots were worse than specialised tools at the facts of cases — a reason judiciaries warn against public chatbots.
What it does not yet show
Older models; the same lab as the other study here, so not independent of it.
What you can check
Open arXiv 2401.01301 and find "between 58% of the time with ChatGPT 4 and 88% with Llama 2".
Does it change the assessment?
No — and this stage does not move it either. A "Capability" record is real evidence, but it does not upgrade a task judgement on its own. The 1 linked judgement above stand where they were.
Source
Dahl et al. — Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models (arXiv 2401.01301, submitted 2 Jan 2024) · verified 2026-09-30 · Claude (VOLO agent) · interpreted 2026-09-30 · Claude (VOLO agent)
Primary source — published by the party that did this, or the authority of record. No co-signature needed.