LO
CapabilityCognitive automation2024-08-23
Multimodal AI models struggled to retrieve the relevant rules, recognise components in CAD images and analyse engineering drawings on DesignQA, a benchmark built on Formula SAE rules
Mechanical engineeroccupation page →
Event date / reported
2024-08-23
Evidence stage
CapabilityA demo, benchmark or paper shows the task can be done. Updates what the technology can do — not what employers will do.
Tasks this bears on
Prototypes, testing and failure investigation
Building and testing prototypes, and investigating why equipment fails in service.
Still human-led≈ Platform inference
Where this applies
A benchmark built on the Formula SAE student competition's rule document, with CAD images and engineering drawings from the MIT Motorsports team. It reports that the models tested, while promising, struggle to reliably retrieve relevant rules, face challenges in recognising technical components in CAD images and encounter difficulty in analysing engineering drawings; even given the full rule document, GPT-4o could not reliably extract specific rules. It tests 2024 models on one competition's rules, and several authors work for a design-software company.
What this means
Reading drawings and rules — the ground under design review and failure investigation — is where these models are weakest. The engineer who reads the drawing against the requirement is not the part being automated.
What it does not yet show
2024 models on one competition's documents; it does not measure testing or failure-investigation work.
What you can check
Open arXiv 2404.07917 (DesignQA) and find "encounter difficulty in analyzing engineering drawings".
Does it change the assessment?
No — and this stage does not move it either. A "Capability" record is real evidence, but it does not upgrade a task judgement on its own. The 1 linked judgement above stand where they were.
Source
Doris et al. (MIT, Autodesk Research) — DesignQA: A Multimodal Benchmark for Evaluating Large Language Models' Understanding of Engineering Documentation, arXiv 2404.07917 (v2, 23 Aug 2024) · verified 2026-09-29 · Claude (VOLO agent) · interpreted 2026-09-29 · Claude (VOLO agent)
Primary source — published by the party that did this, or the authority of record. No co-signature needed.