LO
DeploymentCognitive automation2025-01-23
In the 2024 US federal AI inventory, of 76 AI systems in operation that described their monitoring, 47 relied on manual model updates and only 5 had a full pipeline with drift detection
Machine learning engineeroccupation page →
Event date / reported
2025-01-23
Evidence stage
DeploymentAn employer has put it into production. Can move the baseline — weighted by scale and how similar the setting is.
Tasks this bears on
When it quietly stops working
Drift, a changed upstream input, a seasonal pattern the training data never saw — and the decision to retrain, roll back, or turn it off.
Being augmented✓ Evidence-backed
Where this applies
The US government's own inventory of AI use across agencies, 2,133 entries. It asks whether there is a process to monitor an AI system's performance after deployment, with four answer options. Counted here from the file: of the 874 entries in Operation and Maintenance, 76 chose one of the four — 47 'Intermittent and Manually Updated' (data science teams work with DevOps engineers to manually update models at scheduled intervals and create metrics to detect distribution shift), 15 'Automated and Regularly Scheduled Updates' (some monitoring and retraining automated, but data science teams still significantly involved), 5 'Established Process of Machine Learning Operations' (automated testing, drift detection and pipeline-driven retraining), and 9 with no monitoring protocols. The other 798 left the question blank or wrote not applicable, and the answers were largely voluntary and skewed towards a few agencies, so this describes the systems that answered, not federal AI as a whole.
What this means
Watching whether a deployed model has quietly stopped working is still mostly people's work in the systems that reported it: most relied on engineers updating models by hand, and fully automated drift detection and retraining was rare.
What it does not yet show
It covers only the federal systems that answered, most answers were voluntary, and the options describe processes, not how well they work; it says nothing about the private sector.
What you can check
Open the 2024 Federal AI Use Case Inventory raw v2 CSV, filter Stage of Development to Operation and Maintenance, and count the answers to the post-deployment monitoring question.
Does it change the assessment?
No. The impact index is never moved by a single event. What this record did: the 1 linked task judgement above now rest on evidence instead of inference.
Source
Office of Management and Budget — 2024 Federal Agency AI Use Case Inventory, consolidated raw data v2 (GitHub, committed 23 January 2025) · verified 2026-09-27 · Claude (VOLO agent) · interpreted 2026-09-27 · Claude (VOLO agent)
Primary source — published by the party that did this, or the authority of record. No co-signature needed.