From Runbook to Robot: Executable Runbooks as Code

From Runbook to Robot: Executable Runbooks as Code

Your wiki says "drain the node" but not which flags this cluster needs or what to check first. Prose runbooks drift, and at 3 a.m. the drift becomes the incident. Executable runbooks turn each step into reviewed code with checks built in. You will build a YAML format, a small Python executor with dry-run and approval gates, and an LLM prompt that converts old wiki pages.

Why prose runbooks drift, and how executable runbooks fix it

A runbook is the written procedure for a known task, such as draining a node. The wiki version is written once, after an incident, while the system keeps changing. Nobody is paged when a page goes stale.

In the diagram, the wiki page still uses --delete-local-data, a flag kubectl renamed to --delete-emptydir-data, and leaves "check it looks healthy" to the reader. The code version lives in Git, is reviewed like any change, and runs through an executor that checks the system before and after.

Before and after: a wiki runbook with a renamed flag, a vague health check and no testing, versus the same runbook as YAML in Git with pre-checks, a dry run, approval and post-checks
Prose drifts silently; code drifts loudly, as a failing check or a diff.

The checks, and the decision about when a human must say yes, move from the reader's head into the file. A post-mortem action such as "update the runbook" becomes a pull request, which is what a post-mortem culture that works needs.

On the toil audit ladder this is rung two: a script a human runs. You need Python 3.9+, PyYAML (pip install pyyaml), kubectl on a test cluster, and an LLM CLI that reads standard input, such as claude -p.

Write the runbook as code in YAML

Keep the format small enough to review. A runbook declares parameters, read-only pre-checks, steps and post-checks. A check is a command plus its exact expected output; a step can add a safer dry-run command and an approval gate. Save this as runbooks/drain-node.yaml:

name: drain-node
version: 3                  # bump on every change
params: [node]              # filled from --param node=...
pre_checks:
  - name: node is Ready
    run: kubectl get node {node} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'
    expect: "True"
  - name: no other node is cordoned
    run: kubectl get nodes --field-selector spec.unschedulable=true -o name | wc -l
    expect: "0"
steps:
  - name: drain
    run: kubectl drain {node} --ignore-daemonsets --delete-emptydir-data --timeout=300s
    dry_run: kubectl drain {node} --ignore-daemonsets --delete-emptydir-data --dry-run=server
    approval: true          # a human must type yes
post_checks:
  - name: node is unschedulable
    run: kubectl get node {node} -o jsonpath='{.spec.unschedulable}'
    expect: "true"

{node} comes from the command line, so one file serves every node. The first pre-check uses kubectl's JSONPath support to print only the Ready condition. The second counts cordoned (unschedulable) nodes, so you never drain two at once.

--dry-run=server sends the request to the API server, which validates it and runs admission control but saves nothing. The classic mistake is an unquoted expect: YAML reads true as a boolean, so the check compares True with true and fails.

A tiny Python executor with dry-run and approval gates

The executor is deliberately boring: under 40 lines of standard library plus PyYAML. The flowchart shows the order of operations and who acts at each point.

Flowchart: load the runbook, abort if pre-checks fail, run only dry_run commands when --dry-run is set, wait for a human to type yes on gated steps, run each step and loop, then finish or stop and page on-call depending on post-checks
Every stop exits non-zero; only a clean run reaches exit 0.

Cyan boxes run unattended and the green diamond needs a person. Every stop exits non-zero, so a CI job or chat bot can tell a clean run from an abort. Start runbook.py with two helpers:

import argparse, shlex, subprocess, sys
import yaml

def sh(cmd, params):
    for key, value in params.items():  # quote values: no shell injection
        cmd = cmd.replace("{" + key + "}", shlex.quote(value))
    print(f"  $ {cmd}")
    res = subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=600)
    return res.returncode, (res.stdout or res.stderr).strip()

def check(kind, items, params):
    for c in items:
        out = sh(c["run"], params)[1]
        ok = out == str(c["expect"])
        print(f"[{kind}] {c['name']}: " + ("ok" if ok else f"FAIL, got {out!r}"))
        if not ok:
            sys.exit(2)

sh() wraps each parameter in shlex.quote, so worker-3; reboot stays one harmless argument. It echoes every command first, which doubles as an audit trail. check() compares trimmed output with expect and exits with code 2 on the first mismatch.

Order matters, because a pipe hides kubectl's exit code: with the cluster unreachable, wc -l still prints 0. The Ready check runs first and fails loudly instead. Then add the main block:

ap = argparse.ArgumentParser()
ap.add_argument("runbook")
ap.add_argument("--param", action="append", default=[])  # repeatable key=value
ap.add_argument("--dry-run", action="store_true")
args = ap.parse_args()
with open(args.runbook) as f:
    rb = yaml.safe_load(f)  # no arbitrary Python objects
params = dict(p.split("=", 1) for p in args.param)

print(f"runbook {rb['name']} v{rb['version']} dry_run={args.dry_run}")
check("pre", rb.get("pre_checks", []), params)
for step in rb["steps"]:
    cmd = step.get("dry_run", "echo no dry_run") if args.dry_run else step["run"]
    gated = step.get("approval") and not args.dry_run
    if gated and input(f"run '{step['name']}'? type yes: ") != "yes":
        sys.exit("not approved, stopping")
    code, out = sh(cmd, params)
    print(out)
    if code:
        sys.exit(f"step '{step['name']}' failed with exit {code}")
if not args.dry_run:
    check("post", rb.get("post_checks", []), params)
print("done")

In dry-run mode each step runs its dry_run command, a failing dry run still exits non-zero, and post-checks are skipped because nothing changed. Try it against a test cluster:

$ python3 runbook.py runbooks/drain-node.yaml --param node=worker-3 --dry-run
runbook drain-node v3 dry_run=True
[pre] node is Ready: ok
[pre] no other node is cordoned: ok
node/worker-3 cordoned (server dry run)
node/worker-3 drained (server dry run)
done

Command echoes and per-pod lines are trimmed here. If another node is already cordoned, you get FAIL, got '1' and exit code 2 before anything changes.

When I guard-railed automated upgrades for a 2000+ machine fleet, the same ideas made host draining safe: check first, limit how many hosts change at once, and verify afterwards.

Convert a legacy wiki runbook with an LLM

Converting dozens of wiki pages by hand is classic toil. Large language models (LLMs) turn messy prose into structured text well, but know nothing about your cluster. So the model drafts, a dry run tests and a human reviews, as the diagram shows.

Data flow: a wiki runbook and a versioned prompt go to an LLM with no cluster credentials, which writes draft YAML with REVIEW comments; CI dry-runs it on staging, a human reviews the pull request, and the merged runbook is run by runbook.py against Kubernetes
The LLM only writes a file; the dry run and the reviewer decide what ships.

The model gets no cluster credentials. Its only output is a file that ships through the same review as hand-written YAML. Keep the prompt in Git with a version in its name, because it shapes every draft. Save it as prompts/wiki-to-runbook.v1.txt:

Convert this runbook to YAML with keys name, version, params,
pre_checks, steps, post_checks. Checks are read-only commands with a
quoted "expect". State-changing steps get "dry_run" when the tool has one,
and "approval: true" if they risk downtime. Use {param} placeholders.
Use only commands from the source; mark vague steps "# REVIEW:".
Output only YAML, no Markdown fences.

"Use only commands from the source" stops the model inventing flags. The # REVIEW: marker turns vague prose into a visible question instead of a guessed check. Run the conversion, then test the draft:

claude -p "$(cat prompts/wiki-to-runbook.v1.txt)" < wiki/drain-node.md > runbooks/drain-node.yaml
grep -n "# REVIEW:" runbooks/drain-node.yaml
python3 runbook.py runbooks/drain-node.yaml --param node=worker-3 --dry-run

claude -p runs non-interactively: the prompt is the instruction and the wiki page arrives on standard input. Review starts at the lines grep finds, and the dry run proves the commands parse on staging. Expect YAML close to the file above, plus lines such as # REVIEW: "check it looks healthy".

The usual failure is Markdown fences around the output, which make PyYAML stop with found character '`' that cannot start any token. Tighten the prompt, or strip the first and last lines.

Treat this as a small LLMOps pipeline: pin the model with --model, version the prompt, and before converting fifty pages, eval it on three runbooks you already converted by hand. One page is one model call, so tokens are cheap; reviewer time is the real budget. Clean runbooks also make a better RAG corpus.

Common pitfalls

  • Checks that cannot fail. Pipes hide errors, so put a loud check, such as node Ready, before any | wc -l check.
  • Wrong cluster. Make kubectl config current-context the first pre-check, so a staging run never hits production.
  • Trusting the draft. A dry run catches unknown flags, not wrong logic. A human still reads every step.
  • Approval fatigue. Gate only steps that risk downtime or data loss, or people type yes without reading.

Key takeaways

  • Executable runbooks turn silent drift into failing checks and reviewable diffs.
  • Give every runbook read-only pre-checks, a dry run, gates on risky steps and post-checks.
  • Keep the executor small and loud: quote parameters, echo commands, exit non-zero on doubt.
  • Let an LLM draft conversions, but send every draft through review and a dry run.

Related articles