LO
CapabilityCognitive automation2024-02-27
The strongest model reached 58% accuracy on statistical and causal questions from textbooks and papers, and models struggled to use causal knowledge and data together
Data scientistoccupation page →
Event date / reported
2024-02-27 · reported 2024-06-09
Evidence stage
CapabilityA demo, benchmark or paper shows the task can be done. Updates what the technology can do — not what employers will do.
Tasks this bears on
Designing experiments and reading causes
Setting up the A/B test or the study so the answer can be trusted, and saying whether a change caused the result or merely came with it.
Still human-led≈ Platform inference
Where this applies
An academic benchmark of 411 questions with data sheets from textbooks, online learning materials and academic papers, testing statistical and causal reasoning. It reports that the strongest model, GPT-4, achieves 58% accuracy, and that models have difficulty with data analysis and causal reasoning and struggle to use causal knowledge and provided data simultaneously. It tests 2024-era models on textbook-style questions, not experiments on real products.
What this means
Telling cause from coincidence is where the models tested were weakest — and it is the question most decisions rest on. That part of the data scientist's work is the least assisted by the tools so far.
What it does not yet show
A 2024 benchmark on textbook questions; newer models may do better, and it does not measure real experiments.
What you can check
Open arXiv 2402.17644 (QRData) and find "The strongest model GPT-4 achieves an accuracy of 58%".
Does it change the assessment?
No — and this stage does not move it either. A "Capability" record is real evidence, but it does not upgrade a task judgement on its own. The 1 linked judgement above stand where they were.
Source
Liu et al. — Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data (QRData), arXiv 2402.17644 (submitted 27 Feb 2024; Findings of ACL 2024) · verified 2026-09-29 · Claude (VOLO agent) · interpreted 2026-09-29 · Claude (VOLO agent)
Primary source — published by the party that did this, or the authority of record. No co-signature needed.