LO
CapabilityCognitive automation2025-06-17
State-of-the-art models passed no more than 34% of code-generation problems on a benchmark written by experienced hardware engineers, and verification tasks were particularly difficult
Electrical engineeroccupation page →
Event date / reported
2025-06-17
Evidence stage
CapabilityA demo, benchmark or paper shows the task can be done. Updates what the technology can do — not what employers will do.
Tasks this bears on
Circuit, board and chip design
Designing analog and digital circuits, writing hardware description code, and laying out boards and chips.
Being augmented≈ Platform inference
Where this applies
A benchmark of 783 problems across 13 task categories — RTL generation, verification, debugging, specification alignment and technical questions — authored by experienced hardware engineers, in both non-agentic and agentic formats. State-of-the-art models achieved no more than 34% pass@1 on code generation, and agentic tasks, especially those involving RTL reuse and verification, were particularly difficult. A test set with 2025 models; the authors' employer designs chips.
What this means
On harder, realistic problems, the best models fail most code generation, and verification is harder still.
What it does not yet show
A benchmark with 2025 models; it does not measure how engineers use them.
What you can check
Open arXiv 2506.14074 (CVDP) and find "no more than 34% pass@1 on code generation".
Does it change the assessment?
No — and this stage does not move it either. A "Capability" record is real evidence, but it does not upgrade a task judgement on its own. The 1 linked judgement above stand where they were.
Source
Pinckney et al. (NVIDIA) — Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verification, arXiv 2506.14074 (17 Jun 2025) · verified 2026-09-30 · Claude (VOLO agent) · interpreted 2026-09-30 · Claude (VOLO agent)
Primary source — published by the party that did this, or the authority of record. No co-signature needed.