DevOps engineer / platform engineer / SRE

Google describing its own operations rather than selling into somebody else, written by a Distinguished Software Engineer and a Distinguished Site Reliability Engineer and published on the company blog. What it states in the past or present tense, which is the half worth recording: the SRE team has developed agents that monitor and improve playbooks from their use during incidents and can generate new playbooks from incidents; an alerting agent that groups, pre-processes and enriches alerts before autonomous handlers take them; agents that consolidate the chat spaces, videos and tracking documents used during an incident, create handoff documents between SREs, draft postmortems, and manage internal and external incident communications; agents created to investigate incidents and, in the words of the post, in some cases to autonomously mitigate issues; and AI Insights, a system that reviews past incidents and feeds what it extracts back to those agents. The rest of the post is plans, and is not recorded. Three limits, and the first is the largest. There is not one number in it: no share of incidents touched, no count of agents running, no before and after, and nothing about how many people are on the rota. Second, the stake is direct and is not hidden by the first-party framing — every component named underneath is a Google product (Gemini, the Agent Development Kit, the Gemini Enterprise Agent Platform, MCP on Google API infrastructure), so this post doubles as a reference architecture for things the publisher sells. Third, it is one operator, and an unusual one: an organisation with twenty years of its own reliability discipline is not a fair stand-in for a team of four keeping a pager. What makes it evidence about this task rather than about tooling is where the agents sit: the work around the decision at three in the morning, assembling the context, handing over, telling people what is happening, writing it up afterwards, is the part that moved, and the post says an agentic approach does not necessarily imply removing people from the process for higher-risk services.

An employer has put it into production. Can move the baseline — weighted by scale and how similar the setting is.

One company, one edge router in Miami, 25 minutes (20:25-20:50 UTC). What the automation authored is the point: a change pushed through Cloudflare's policy automation platform removed prefix-list references and left a policy term so permissive that it marked every internal prefix as acceptable to advertise. A human reverted it and paused automation on that router. Recorded as deployment rather than constraint because adoption was not suppressed: the remediation Cloudflare states is adding automatic routing policy evaluation into its CI/CD pipelines — a machine checking the machine, not a human gate. It establishes that at one large network operator, engineers make routing policy changes through an automation platform that generates and applies the configuration in production; it does not say what share of that operator's configuration is machine-written, and one incident at one provider is not a measure of the industry.

An employer has put it into production. Can move the baseline — weighted by scale and how similar the setting is.

A practitioner survey, so the 90% is self-reported use rather than measured use, and the accompanying '80% believe it increased their productivity' is a belief and must be read as one — this site holds a randomised trial in which experienced developers were 19% slower while believing they were 20% faster, so belief about one's own speed is not evidence about speed. What is not self-report about one's own performance is the correlation the researchers compute across respondents, and it is the useful part: adoption relates positively to throughput and product performance and negatively to delivery stability. The mechanism the authors give is specific — without strong automated testing, mature version control and fast feedback loops, higher change volume produces instability, and teams in loosely coupled architectures see gains while tightly coupled ones see little or none. Google Cloud publishes this and sells the tools it asks about.

Measured, large-scale use of a tool for real work, where the decision to use it was the worker's rather than an employer's. It is more than a capability record — the work is real, not a demo — and less than a deployment record, because no employer put it into production, required it, or built a process around it. Weighted `cautious`: `automating` means the machine can do the task AND there are adoption signs, and this is an adoption sign — but usage can be experimental, and much of the measurement comes from a party with a stake, so one record is never enough and two independent ones are. Note who is counting. Vendor telemetry sees this directly and sells the tool, so such a record names that stake in its scope; a statistics agency asking firms whether their workers use AI in tasks sees the same channel with no stake at all, and that is the better source where it exists.

The cloud provider's own incident report on a multi-hour outage across many of its services. It says the change at the root did not have appropriate error handling nor was it feature-flag protected; that the service lacked randomized exponential backoff, so recovery took up to about 2 hours 40 minutes in one region; and that the remedies include modularizing the service's architecture so functionality is isolated and fails open, and propagating data replication incrementally with time to validate. It is one company's account of its own outage; the failed piece was ordinary code, not AI.

Failure, rollback, regulation or cost is suppressing adoption. Can lower an assessment or widen its uncertainty.

The security vendor's own root cause analysis of the July 2024 update that crashed Windows machines worldwide. It finds that the automated content validator contained a logic error; that for automated testing a static set of 12 test cases was selected; and that each template instance should be deployed in a staged rollout, with two independent third-party software security vendors engaged to review further. It is the company's account of its own failure; the automation that failed was a validator, not AI.

Failure, rollback, regulation or cost is suppressing adoption. Can lower an assessment or widen its uncertainty.

One US federal agency. The case study, published by the General Services Administration with OPM's deputy CIO as contact, says OPM's Azure cloud spend grew over 2,000% since 2021, which led it to a dedicated team for monitoring and managing cloud costs; that its anomaly detection establishes dashboards to detect, identify and act on unexpected increases in cloud spend; that it uses a decision matrix for each bucket of decisions; and that the team tracks decisions such as reserved-instance commitments, rightsizing, budget management and forecasting. It does not say whether the detection uses AI or simple thresholds. Progress is reported as of March 2024; dated to the end of that month.

An employer has put it into production. Can move the baseline — weighted by scale and how similar the setting is.

A commissioned sector study, not a measurement, and it is recorded here because it points the other way from most of this base. The same document that expects stand-alone support engineering to shrink expects the work to land on this role: it says the DevOps function will take over overseeing service level agreements and developing new systems, and that the development function will take over managing and optimising system performance, bringing development and operations together to provide more holistic support, and it lists automation and orchestration engineers as a low-impact role that high-impact infrastructure engineers could move into, without saying demand for it is growing. It also names DevOps Engineer as the destination it considers an easy or moderate move for the support engineers it expects to be displaced. None of that is evidence that anyone was hired, that headcount rose, or that the transition happened — it is one agency's expectation, published before generative AI reached the public, covering Singapore only and built from stakeholder interviews rather than employment data. What it does establish is that the displacement finding elsewhere in the same report is not a claim that the work disappears.

A named person with standing publicly predicted something, on a date, in an attributable statement. It is recorded so that who said what, and when, stays checkable — and it never moves a task's assessment, because a prediction is not an observation. Its value arrives later: the record sits on the same page as the evidence about that occupation, so anyone reading the forecast reads the record of what happened next beside it. That is the reckoning; this site publishes no verdict on whether a forecast came true.

The rung that used to hire juniors — writing and wiring configuration — is the one with the clearest mechanism pointed at it, because it is code with a machine-checkable outcome. What that leaves for a beginner is the on-call, which is the hardest part and the one you were supposed to earn. Push to be in incident reviews long before you are on the rota.

Your leverage is the design decision and the incident call, and both are small slices of the week. The risk is not that they get automated, it is that the hours around them get thinned until one engineer covers services no one person can hold in their head. Watch services-per-engineer the way a waiter watches covers-per-server.

Stay and strengthen

Move toward design and away from wiring

The design decision has no automatic referee, which is exactly why no tool can close the loop on it — and it is the part of the job that compounds.

Reshape the role

Own the operability of what others build

More configuration being generated means more systems nobody is accountable for, and somebody has to hold the standard for what may be run in production.

The configuration half is moving and moving fast, because infrastructure code either applies cleanly or errors — a machine-checkable outcome is what lets a model retry unattended. The on-call is not, because an incident is by definition the failure nobody anticipated and the call is about acceptable damage rather than a correct answer. The likely shape is not removal but thinning: fewer engineers covering more services, which keeps every task and makes the rota worse.

No date. Two numbers you can produce yourself say more than any forecast: what share of your week goes to work where applying it either succeeds or errors, and how many services each engineer on your rota is now responsible for. The first is your task exposure; the second is the change that actually reaches people in this job, and it moves quietly after tooling lands.

Not usually, and this is the clearest case on the site of a tool that looks like a saving and behaves like a debt. Every resource created is a resource somebody has to keep, secure and eventually delete, so faster configuration produces more configuration rather than less work — the hours move from writing to untangling. Untangling is invisible on a roadmap, which is exactly why it gets budgeted as a saving.

It is the safest task in the occupation and also the smallest. Whether a design was right is discovered long after it is made, which means there is no automatic referee and therefore no loop a tool can close — but design decisions take a small share of most teams' hours. A role can be secure in its most senior task and still lose most of its hours, and that gap is the thing to plan around.