From Runbook to Robot: Executable Runbooks as Code

Your wiki says "drain the node" but not which flags this cluster needs or what to check first. Prose runbooks drift, and at 3 a.m. the drift becomes the incident. Executable runbooks turn each step into reviewed code with checks built in. You will build a YAML format, a small Python executor with dry-run and approval gates, and an LLM prompt that converts old wiki pages.
Why prose runbooks drift, and how executable runbooks fix it
A runbook is the written procedure for a known task, such as draining a node. The wiki version is written once, after an incident, while the system keeps changing. Nobody is paged when a page goes stale.
In the diagram, the wiki page still uses --delete-local-data, a flag kubectl renamed to --delete-emptydir-data, and leaves "check it looks healthy" to the reader. The code version lives in Git, is reviewed like any change, and runs through an executor that checks the system before and after.
The checks, and the decision about when a human must say yes, move from the reader's head into the file. A post-mortem action such as "update the runbook" becomes a pull request, which is what a post-mortem culture that works needs.
On the toil audit ladder this is rung two: a script a human runs. You need Python 3.9+, PyYAML (pip install pyyaml), kubectl on a test cluster, and an LLM CLI that reads standard input, such as claude -p.
Write the runbook as code in YAML
Keep the format small enough to review. A runbook declares parameters, read-only pre-checks, steps and post-checks. A check is a command plus its exact expected output; a step can add a safer dry-run command and an approval gate. Save this as runbooks/drain-node.yaml:
name: drain-node
version: 3 # bump on every change
params: [node] # filled from --param node=...
pre_checks:
- name: node is Ready
run: kubectl get node {node} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'
expect: "True"
- name: no other node is cordoned
run: kubectl get nodes --field-selector spec.unschedulable=true -o name | wc -l
expect: "0"
steps:
- name: drain
run: kubectl drain {node} --ignore-daemonsets --delete-emptydir-data --timeout=300s
dry_run: kubectl drain {node} --ignore-daemonsets --delete-emptydir-data --dry-run=server
approval: true # a human must type yes
post_checks:
- name: node is unschedulable
run: kubectl get node {node} -o jsonpath='{.spec.unschedulable}'
expect: "true"
{node} comes from the command line, so one file serves every node. The first pre-check uses kubectl's JSONPath support to print only the Ready condition. The second counts cordoned (unschedulable) nodes, so you never drain two at once.
--dry-run=server sends the request to the API server, which validates it and runs admission control but saves nothing. The classic mistake is an unquoted expect: YAML reads true as a boolean, so the check compares True with true and fails.
A tiny Python executor with dry-run and approval gates
The executor is deliberately boring: under 40 lines of standard library plus PyYAML. The flowchart shows the order of operations and who acts at each point.
Cyan boxes run unattended and the green diamond needs a person. Every stop exits non-zero, so a CI job or chat bot can tell a clean run from an abort. Start runbook.py with two helpers:
import argparse, shlex, subprocess, sys
import yaml
def sh(cmd, params):
for key, value in params.items(): # quote values: no shell injection
cmd = cmd.replace("{" + key + "}", shlex.quote(value))
print(f" $ {cmd}")
res = subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=600)
return res.returncode, (res.stdout or res.stderr).strip()
def check(kind, items, params):
for c in items:
out = sh(c["run"], params)[1]
ok = out == str(c["expect"])
print(f"[{kind}] {c['name']}: " + ("ok" if ok else f"FAIL, got {out!r}"))
if not ok:
sys.exit(2)
sh() wraps each parameter in shlex.quote, so worker-3; reboot stays one harmless argument. It echoes every command first, which doubles as an audit trail. check() compares trimmed output with expect and exits with code 2 on the first mismatch.
Order matters, because a pipe hides kubectl's exit code: with the cluster unreachable, wc -l still prints 0. The Ready check runs first and fails loudly instead. Then add the main block:
ap = argparse.ArgumentParser()
ap.add_argument("runbook")
ap.add_argument("--param", action="append", default=[]) # repeatable key=value
ap.add_argument("--dry-run", action="store_true")
args = ap.parse_args()
with open(args.runbook) as f:
rb = yaml.safe_load(f) # no arbitrary Python objects
params = dict(p.split("=", 1) for p in args.param)
print(f"runbook {rb['name']} v{rb['version']} dry_run={args.dry_run}")
check("pre", rb.get("pre_checks", []), params)
for step in rb["steps"]:
cmd = step.get("dry_run", "echo no dry_run") if args.dry_run else step["run"]
gated = step.get("approval") and not args.dry_run
if gated and input(f"run '{step['name']}'? type yes: ") != "yes":
sys.exit("not approved, stopping")
code, out = sh(cmd, params)
print(out)
if code:
sys.exit(f"step '{step['name']}' failed with exit {code}")
if not args.dry_run:
check("post", rb.get("post_checks", []), params)
print("done")
In dry-run mode each step runs its dry_run command, a failing dry run still exits non-zero, and post-checks are skipped because nothing changed. Try it against a test cluster:
$ python3 runbook.py runbooks/drain-node.yaml --param node=worker-3 --dry-run
runbook drain-node v3 dry_run=True
[pre] node is Ready: ok
[pre] no other node is cordoned: ok
node/worker-3 cordoned (server dry run)
node/worker-3 drained (server dry run)
done
Command echoes and per-pod lines are trimmed here. If another node is already cordoned, you get FAIL, got '1' and exit code 2 before anything changes.
When I guard-railed automated upgrades for a 2000+ machine fleet, the same ideas made host draining safe: check first, limit how many hosts change at once, and verify afterwards.
Convert a legacy wiki runbook with an LLM
Converting dozens of wiki pages by hand is classic toil. Large language models (LLMs) turn messy prose into structured text well, but know nothing about your cluster. So the model drafts, a dry run tests and a human reviews, as the diagram shows.
The model gets no cluster credentials. Its only output is a file that ships through the same review as hand-written YAML. Keep the prompt in Git with a version in its name, because it shapes every draft. Save it as prompts/wiki-to-runbook.v1.txt:
Convert this runbook to YAML with keys name, version, params,
pre_checks, steps, post_checks. Checks are read-only commands with a
quoted "expect". State-changing steps get "dry_run" when the tool has one,
and "approval: true" if they risk downtime. Use {param} placeholders.
Use only commands from the source; mark vague steps "# REVIEW:".
Output only YAML, no Markdown fences.
"Use only commands from the source" stops the model inventing flags. The # REVIEW: marker turns vague prose into a visible question instead of a guessed check. Run the conversion, then test the draft:
claude -p "$(cat prompts/wiki-to-runbook.v1.txt)" < wiki/drain-node.md > runbooks/drain-node.yaml
grep -n "# REVIEW:" runbooks/drain-node.yaml
python3 runbook.py runbooks/drain-node.yaml --param node=worker-3 --dry-run
claude -p runs non-interactively: the prompt is the instruction and the wiki page arrives on standard input. Review starts at the lines grep finds, and the dry run proves the commands parse on staging. Expect YAML close to the file above, plus lines such as # REVIEW: "check it looks healthy".
The usual failure is Markdown fences around the output, which make PyYAML stop with found character '`' that cannot start any token. Tighten the prompt, or strip the first and last lines.
Treat this as a small LLMOps pipeline: pin the model with --model, version the prompt, and before converting fifty pages, eval it on three runbooks you already converted by hand. One page is one model call, so tokens are cheap; reviewer time is the real budget. Clean runbooks also make a better RAG corpus.
Common pitfalls
- Checks that cannot fail. Pipes hide errors, so put a loud check, such as node Ready, before any
| wc -lcheck. - Wrong cluster. Make
kubectl config current-contextthe first pre-check, so a staging run never hits production. - Trusting the draft. A dry run catches unknown flags, not wrong logic. A human still reads every step.
- Approval fatigue. Gate only steps that risk downtime or data loss, or people type
yeswithout reading.
Key takeaways
- Executable runbooks turn silent drift into failing checks and reviewable diffs.
- Give every runbook read-only pre-checks, a dry run, gates on risky steps and post-checks.
- Keep the executor small and loud: quote parameters, echo commands, exit non-zero on doubt.
- Let an LLM draft conversions, but send every draft through review and a dry run.
Related articles

AI Incident Triage: A Co-Pilot for the First 10 Minutes
Wire Alertmanager to a small Python worker that queries Prometheus, your deploy log and a RAG index of post-mortems, then posts a cited triage summary to Slack. Read-only by design: humans approve every change.

AIOps Without the Hype: What AI Improves in Observability
Anomaly detection, log clustering, natural-language queries and autonomous RCA: which AIOps features work today, the data quality they need, and how an SRE team can adopt them in 30 days.

RAG 101: Build Your First RAG Pipeline over Runbooks
Turn a folder of Markdown runbooks into an on-call assistant: chunk by heading, embed locally with Chroma, answer with Claude, cite every source, and refuse when no runbook matches.