LO
CapabilityCognitive automation2026-04-24
On Cornetto, a benchmark of 231 network configuration repair problems, the best of nine language models fully resolved only 25% of scenarios
Network engineeroccupation page →
Event date / reported
2026-04-24
Evidence stage
CapabilityA demo, benchmark or paper shows the task can be done. Updates what the technology can do — not what employers will do.
Tasks this bears on
Configuration and changes
Making configuration changes to routers, switches, firewalls and radio networks, and managing them safely.
Automating✓ Evidence-backed
Where this applies
A benchmark of 231 problems for fixing configurations on network topologies of 20 to 754 nodes, checked by formal verification, on which nine language models were evaluated. The paper reports that the best-performing model resolved only 25% of scenarios, that strictly correct resolutions — every fault fixed with no regressions — occurred in at most 25.5% of cases, and that models are best deployed as human-in-the-loop assistants. It is a preprint on synthesised scenarios.
What this means
Routine changes can be automated; repairing a broken configuration across a large network is still mostly beyond language models. The complex fix — and the check that it breaks nothing else — stays with the engineer.
What it does not yet show
A preprint benchmark on synthesised scenarios; it does not measure real operations.
What you can check
Open arXiv 2604.22513 and find "resolved only 25% of scenarios" in the paper.
Does it change the assessment?
No — and this stage does not move it either. A "Capability" record is real evidence, but it does not upgrade a task judgement on its own. The 1 linked judgement above stand where they were.
Source
Protogeros, Asadli, Hoffman, Vanbever (ETH Zürich) — Benchmarking LLM-Driven Network Configuration Repair, arXiv 2604.22513 (submitted 24 Apr 2026) · verified 2026-09-29 · Claude (VOLO agent) · interpreted 2026-09-29 · Claude (VOLO agent)
Primary source — published by the party that did this, or the authority of record. No co-signature needed.
This record is cited in