The Automation Operations Manual

An automation in production is a living dependency: upstream systems change formats, volumes shift, models update, and the business quietly evolves the process the software encodes. None of this announces itself. The difference between automations that compound for years and automations that rot in six months is not build quality — it's whether anyone is running them.

Running an automation is a light but real job. This manual is that job, written down: the rhythms, the signals, and the plays. It pairs with SCH-04 (the instrumentation) and DSP-14 (the argument for budgeting it).

01The operating rhythms

• ▸Weekly: the fifteen-minute review. Same time, same three questions: did volume move and do we know why; what escalated to humans and were the escalations right; pull five random outputs — would a competent teammate have done the same? Plus one ladder decision: does anything deserve promotion or demotion on the trust ladder?
• ▸Monthly: the numbers pass. The three ROI numbers (volume, quality, outcome — per DSP-12) against baseline, the cost line (model usage, review minutes), and the miss log review. Twenty minutes, same slide, every month.
• ▸Quarterly: the health check. The full inspection: eval suite re-run, access review (who can touch what, still correct?), documentation drift check, dependency scan (what upstream changed this quarter?), and the keep/change/kill decision made explicitly.

02The drift signals, ranked

Drift is the failure mode with no error message. These are the signals worth watching, in the order they usually fire:

• ▸Human override rate rising. The earliest and most reliable alarm. Reviewers correcting more means quality falling — weeks before anyone complains. Alert when it doubles against its four-week median.
• ▸Escalation-lane share shifting. More items routing to humans means inputs changed shape or confidence broke. Fewer can be worse: silent auto-approval of what used to get checked.
• ▸Input distribution moving. Token-length and type distributions of incoming work are a cheap proxy for "the world changed." A new document format upstream shows up here first.
• ▸Volume diverging from the business. If sales grew 20% and automation runs didn't, users are quietly routing around the system. Volume flat against a growing business is abandonment in progress.
• ▸Latency creeping. Usually a symptom of retries, which are a symptom of something upstream degrading. Latency trends are free to watch and often the only visible smoke.

03The incident play

When an automation misbehaves in production, the play is containment, then diagnosis, then repair — in that order. Containment: demote the affected action types down the trust ladder (auto → queue, queue → gate) or hit the kill switch; the underlying workflow keeps running manually because you designed that fallback (you designed that fallback?). Diagnosis: pull the traces (SCH-04) for the bad outputs and find what changed — input shape, upstream format, model version, prompt regression. Repair: fix, re-run the eval suite, and re-graduate the demoted actions through the ladder rather than flipping them straight back.

Then the part everyone skips: the miss goes in the public log with what changed because of it. Incidents handled visibly build more trust than incident-free months — because the team learns the system is watched.

04Documentation that survives handoffs

Every automation needs a runbook that a competent stranger could operate from: what it does and for whom, the trigger and the systems it touches, the trust-ladder state of each action type, the known failure modes and their plays, the kill switch location, and the current owner. One page, living next to the lane ledger, updated at the quarterly health check.

The runbook is also your insurance against the quiet catastrophe of automation ops: the owner changing jobs. Ownership transfers are when unowned rhythms die — the runbook plus a one-hour handoff session keeps the review cadence alive through the transition.