LO
CapabilityCognitive automation2023-12-10
GPT-4 passed 43.5% of human-written problems on the first try in VerilogEval, a benchmark of 156 hardware-description problems that leaves out power, performance and area
Electrical engineeroccupation page →
Event date / reported
2023-12-10
Evidence stage
CapabilityA demo, benchmark or paper shows the task can be done. Updates what the technology can do — not what employers will do.
Tasks this bears on
Circuit, board and chip design
Designing analog and digital circuits, writing hardware description code, and laying out boards and chips.
Being augmented≈ Platform inference
Where this applies
A benchmark of 156 problems from the Verilog instructional website HDLBits, tested for functional correctness by simulation against a golden solution. In the corrected v2 results, GPT-4 passed 43.5% of the human-written problems on the first try and 58.9% within ten tries; GPT-3.5 passed 26.7%. The authors note that much of hardware design revolves around optimising power, performance and area, which the benchmark does not measure. It tests short instructional problems with 2023 models; the authors' employer designs chips and builds AI for chip design.
What this means
Models can write working hardware code for a minority of textbook-style problems on the first try. The design work that matters most in practice — meeting power, performance and area — is not what this measures, and it stays with the engineer.
What it does not yet show
Short instructional problems with 2023 models; it does not measure real chip projects or current models.
What you can check
Open arXiv 2309.07544 (VerilogEval), read the abstract (156 problems from HDLBits), then Table II in the v2 PDF.
Does it change the assessment?
No — and this stage does not move it either. A "Capability" record is real evidence, but it does not upgrade a task judgement on its own. The 1 linked judgement above stand where they were.
Source
Liu et al. (NVIDIA) — VerilogEval: Evaluating Large Language Models for Verilog Code Generation, arXiv 2309.07544 (v2, 10 Dec 2023) · verified 2026-09-29 · Claude (VOLO agent) · interpreted 2026-09-29 · Claude (VOLO agent)
Primary source — published by the party that did this, or the authority of record. No co-signature needed.