Agents on call: DevOps, QA and incidents without a rota

Building a product is a project. Running it is a payroll line that never ends. For a small SaaS the operations burden — monitoring, deploys, incidents, the on-call rota nobody wants — is usually what forces the first two or three hires, long before feature work does.

This is the part of the stack where agents have moved fastest, because operations work is unusually well suited to them: the inputs are structured, the actions are scriptable, and success is measurable.

What the loop actually does

An alert fires. A triage agent correlates it against recent deploys, error rates and traces, forms a hypothesis, and either acts or escalates. If it acts, it opens a pull request with the patch and the reasoning, which goes through the same review and test gates as any other change. If the change passes, a deploy agent canaries it and watches its own rollout.

The important property is that the loop closes. The thing that raised the alert is the thing that verifies the fix, so a patch that does not actually resolve the symptom does not get quietly marked done.

Deciding what an agent may do unsupervised

Not all remediation is equal, and the useful split is by reversibility rather than by severity.

That last row is not a performance judgement. Some actions should require a human because the cost of being wrong is unbounded, and that is true regardless of how good the agent gets.

The channel is the interface

The owner of an agent-run system should not be watching dashboards. Dashboards are a pull interface, and the whole point is that you are not sitting there. Every state change should arrive as a message in Slack or Discord with a link to the underlying artefact.

• Alert raised — what fired, what changed recently, the agent's current hypothesis.
• Patch opened — a link to the PR, the diff summary, and what evidence led to it.
• Deploy shipped — which environment, which commit, the canary result.
• Rollback fired — this one should be loud, and should say what triggered it.
• Daily digest — everything that resolved itself, so you can see the system working without being interrupted by it.

Route them to separate channels by urgency. A rollback and a nightly digest in the same channel means one of them gets ignored, and it will be the wrong one.

Webhooks are the mechanism and they are unglamorous: an incoming webhook per channel, a formatter that turns an event into a message with a link. The engineering effort is small. The discipline is in deciding what is worth a notification, because a channel that fires forty times a day gets muted within a week, and a muted channel is worse than no channel at all.

QA agents, and what they are actually for

A QA agent checks a deployed build against the specification, not against the code. It drives the real interface, follows the acceptance criteria from the SRS, and reports what it could not do. This catches a category of failure unit tests structurally cannot: the feature works exactly as implemented, and the implementation is not what was asked for.

Run it against staging on every deploy and against production on a schedule. Production QA is how you find the failures that only exist with real data — the ones that never appear in a seeded test environment.

The threshold that keeps it honest

The single most important setting in this system is when the agent stops and pages a human. Set it too high and you are on call anyway; too low and you find out about outages from customers.

Something has to escalate on: an alert the agent cannot correlate to a known pattern, a fix that failed twice, anything touching data or money, and any incident that lasts longer than a set wall-clock time regardless of confidence. That last one matters most — an agent quietly failing to resolve something for forty minutes is the failure mode you will actually meet.

Google's SRE book predates all of this and remains the best description of what you are automating. The DORA research has the metrics worth measuring against — if deploy frequency rises while change failure rate holds, the system is working.

Getting there without a bad month

1. Run the agent in observe-only mode first. It posts what it would have done, and you compare that against what you did. Two weeks of this tells you whether to trust it.
2. Grant the reversible actions: restarts, scaling, rollbacks. These are the highest-value and lowest-risk, and rollback alone removes most of the 3am pages.
3. Let it open PRs for everything else. A patch you review at 9am beats an outage you fix at 4am.
4. Move to unattended merge per service, starting with the least critical one, only once the record is dull.
5. Keep the escalation path permanently. A system that cannot page a human is not more autonomous, it is less safe.

The end state is not zero humans. It is that the human is asleep during most incidents and reads about them over coffee. That is a genuinely different business — it is the difference between needing an on-call rota and not. How small a SaaS team can be works through what that does to headcount, and scale and re-architecture is where we do this work.

Frequently asked questions

References

1. Site Reliability Engineering — Google
2. DORA research programme — DORA
3. Claude Code hooks — Anthropic
4. Model Context Protocol — MCP