LO
CapabilityCognitive automation2025-06-17
State-of-the-art models passed no more than 34% of code-generation problems on a benchmark written by experienced hardware engineers, and verification tasks were particularly difficult
Chip design engineeroccupation page →
Event date / reported
2025-06-17
Evidence stage
CapabilityA demo, benchmark or paper shows the task can be done. Updates what the technology can do — not what employers will do.
Tasks this bears on
Writing RTL
Writing the hardware description code (Verilog, VHDL) that describes each block's logic.
Being augmented≈ Platform inference
Verification and debug
Building testbenches, running simulations and tracking down why a design does not behave as specified.
Being augmented≈ Platform inference
Where this applies
A benchmark of 783 problems across 13 task categories — RTL generation, verification, debugging, specification alignment and technical questions — authored by experienced hardware engineers, in both non-agentic and agentic formats. State-of-the-art models achieved no more than 34% pass@1 on code generation, and agentic tasks, especially those involving RTL reuse and verification, were particularly difficult. A test set with 2025 models; the authors' employer designs chips.
What this means
On harder, realistic problems, the best models fail most code generation, and verification is harder still.
What it does not yet show
A benchmark with 2025 models; it does not measure how engineers use them.
What you can check
Open arXiv 2506.14074 (CVDP) and find "no more than 34% pass@1 on code generation".
Does it change the assessment?
No — and this stage does not move it either. A "Capability" record is real evidence, but it does not upgrade a task judgement on its own. The 2 linked judgements above stand where they were.
Source
Pinckney et al. (NVIDIA) — Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verification, arXiv 2506.14074 (17 Jun 2025) · verified 2026-09-30 · Claude (VOLO agent) · interpreted 2026-09-30 · Claude (VOLO agent)
Primary source — published by the party that did this, or the authority of record. No co-signature needed.