LO
CapabilityCognitive automation2025-04-08
On FEABench, AI models produced executable calls to a commercial simulation tool 88% of the time but almost never computed a result within 10% of the correct answer
Mechanical engineeroccupation page →
Event date / reported
2025-04-08
Evidence stage
CapabilityA demo, benchmark or paper shows the task can be done. Updates what the technology can do — not what employers will do.
Tasks this bears on
Simulation and analysis
Setting up finite-element, thermal and fluid simulations, and judging whether the results are credible.
Being augmented≈ Platform inference
Where this applies
A benchmark of engineering problems that models solve by driving COMSOL Multiphysics, a commercial finite-element tool, through its API. The best strategy generated executable API calls 88% of the time; but the paper says the model passed the strict cut — a valid target within 10% of the correct answer — for a single problem in two of its experiments, and on the gold problems Claude 3.5 Sonnet produced a valid target for 1 of 15. It is a small preprint benchmark with 2024–2025 models, and the authors' employer develops one of the models tested.
What this means
Models can operate simulation software; they rarely get the right number out of it. The engineer's judgement on whether a result is credible is exactly the part this benchmark shows missing.
What it does not yet show
A small preprint benchmark; it does not measure simulation work on real products.
What you can check
Open arXiv 2504.06260 (FEABench) and find "generates executable API calls 88% of the time".
Does it change the assessment?
No — and this stage does not move it either. A "Capability" record is real evidence, but it does not upgrade a task judgement on its own. The 1 linked judgement above stand where they were.
Source
Mudur et al. (Google Research, Harvard) — FEABench: Evaluating Language Models on Multiphysics Reasoning Ability, arXiv 2504.06260 (submitted 8 Apr 2025) · verified 2026-09-29 · Claude (VOLO agent) · interpreted 2026-09-29 · Claude (VOLO agent)
Primary source — published by the party that did this, or the authority of record. No co-signature needed.
This record is cited in