RAG 101: Build Your First RAG Pipeline over Runbooks

Every team has runbooks, and at 3 a.m. nobody can find the right one. A RAG pipeline (retrieval-augmented generation) fixes that: it searches your docs first, then lets an LLM answer only from what it found. In this guide you will build one over a folder of Markdown runbooks in about 50 lines of Python, get answers that cite their sources, and learn what breaks first.
How a RAG pipeline works
A large language model (LLM) only knows its training data, so it has never read your checkout runbook. RAG fixes that at question time: find the passages that match the question, paste them into the prompt, and tell the model to answer from them alone. The diagram shows the six stages on two paths.
The top row runs whenever a runbook changes. It loads each file, chunks it into sections, embeds each chunk and stores the result. An embedding is a list of numbers that places text with similar meaning close together, so you can search by meaning instead of keywords. The bottom row runs per question: it retrieves the nearest chunks and generates an answer that cites them.
You need Python 3.9 or newer, an Anthropic API key and a few Markdown runbooks. Chroma is the vector database: it runs inside your Python process, saves to a local folder and ships a default embedding model (all-MiniLM-L6-v2) that runs on your CPU. Only the question and the matching excerpts leave your machine, sent to Claude through the Anthropic Python SDK.
Step 1: Set up the project
Start with a clean virtual environment. Chroma pulls in ONNX Runtime to run the embedding model locally, and the Anthropic SDK reads your key from an environment variable, so it never lands in code.
python3 -m venv .venv && source .venv/bin/activate
pip install chromadb anthropic
export ANTHROPIC_API_KEY="your-key-here"
ls runbooks/
# cert-rotation.md checkout-5xx.md disk-full.md kafka-consumer-lag.md
Four or five real runbooks are plenty for a first test. If import chromadb later fails with "Chroma requires sqlite3 >= 3.35.0", your Python was built against an old system SQLite. A newer Python from your package manager, pyenv or Homebrew fixes it.
Step 2: Load, chunk and index the runbooks
Chunking decides what the model gets to read, so it matters more than any other setting. Runbooks already have one heading per step, and splitting on those headings keeps each command next to its explanation. The diagram shows one runbook becoming four chunks.
Each chunk starts with the runbook title, so a section called "Roll back" still says which service it belongs to. Its ID and source path are what answers will cite later. Save this as ingest.py:
import pathlib, re
import chromadb
MAX_CHARS = 1000 # the embedder reads only 256 tokens
col = chromadb.PersistentClient(path=".chroma").get_or_create_collection(
name="runbooks", configuration={"hnsw": {"space": "cosine"}})
def chunks(text):
# split on ## and ### headings, not on "# " comments in code
for section in re.split(r"\n(?=#{2,3} )", text):
for i in range(0, len(section), MAX_CHARS):
if piece := section[i:i + MAX_CHARS].strip():
yield piece
for path in sorted(pathlib.Path("runbooks").glob("**/*.md")):
text = path.read_text(encoding="utf-8")
title = text.splitlines()[0].lstrip("# ")
parts = list(chunks(text))
col.delete(where={"source": str(path)}) # drop stale chunks
col.upsert(
ids=[f"{path.name}#{n}" for n in range(len(parts))],
documents=[f"{title}\n{p}" for p in parts],
metadatas=[{"source": str(path), "chunk": n} for n in range(len(parts))])
print(f"{path}: {len(parts)} chunks")
print("total chunks:", col.count())
PersistentClient keeps the index in .chroma/, and the cosine setting makes distances easy to read: 0 means the same meaning, larger means further apart. The delete before the upsert matters: without it, a runbook that shrinks from six sections to four keeps two stale chunks. The first run downloads the embedding model into ~/.cache/chroma, then prints a line per file:
$ python ingest.py
runbooks/cert-rotation.md: 3 chunks
runbooks/checkout-5xx.md: 4 chunks
runbooks/disk-full.md: 3 chunks
runbooks/kafka-consumer-lag.md: 5 chunks
total chunks: 15
A file that reports one chunk has no level-two headings, so it was cut blindly every 1,000 characters. Add headings rather than tuning the splitter, and re-run the script from CI whenever the runbooks change.
Step 3: Retrieve and generate a cited answer
Now the query path. The flowchart below shows its two decisions. The first matters most: if nothing in the index is close to the question, the script refuses instead of letting the model improvise.
The model never acts on its own. It drafts, and a human checks the citations before running anything. Both "no" branches end in a docs fix, so every failure improves the runbooks. Save this as ask.py:
import os, sys
import anthropic, chromadb
MODEL = os.environ.get("RAG_MODEL", "claude-sonnet-5-5") # pin and version it
MAX_DISTANCE = 0.6 # tune on your own questions
question = " ".join(sys.argv[1:])
col = chromadb.PersistentClient(path=".chroma").get_collection(name="runbooks")
hits = col.query(query_texts=[question], n_results=4) # one result list per query
context = [(f"{m['source']}#{m['chunk']}", doc) for doc, m, dist in
zip(hits["documents"][0], hits["metadatas"][0], hits["distances"][0])
if dist <= MAX_DISTANCE]
if not context:
sys.exit("No runbook covers this. Escalate to a human.")
excerpts = "\n\n".join(f"[{i}] {src}\n{doc}" for i, (src, doc) in enumerate(context, 1))
reply = anthropic.Anthropic().messages.create( # reads ANTHROPIC_API_KEY
model=MODEL, max_tokens=500,
system="Answer only from the numbered runbook excerpts. Cite them like [1]. "
"If they do not contain the answer, say so.",
messages=[{"role": "user", "content": f"{excerpts}\n\nQuestion: {question}"}])
print(reply.content[0].text, "\n\nSources:")
for i, (src, _) in enumerate(context, 1):
print(f" [{i}] {src}")
print(f"tokens in/out: {reply.usage.input_tokens}/{reply.usage.output_tokens}")
n_results=4 fetches the four nearest chunks, and the distance cut-off drops weak matches. If none survive, sys.exit prints the refusal and exits with status 1 before a single token is spent. The system prompt limits Claude to the numbered excerpts and asks for citations. RAG_MODEL lets you swap the model without a code change, and the last line prints token usage: your cost per request.
Run it: ask a question, check the sources
Ask one real on-call question and one the runbooks cannot answer. The output is illustrative; yours depends on your runbooks.
$ python ask.py "checkout returns 5xx after a deploy, what first?"
Check the latest rollout with kubectl rollout history
deployment/checkout -n shop [1]. If a revision landed just
before the errors, roll back with kubectl rollout undo [2].
Sources:
[1] runbooks/checkout-5xx.md#1
[2] runbooks/checkout-5xx.md#2
tokens in/out: 912/74
$ python ask.py "how do I reset the payroll admin password?"
No runbook covers this. Escalate to a human.
Check two things in every answer: each claim carries a citation, and each cited chunk really says what the answer claims. The refusal is a feature, because it names a runbook you are missing. If good questions get refused, raise the cut-off slightly; if off-topic questions get answers, lower it.
From script to service: evals, versions and budgets
I have built RAG pipelines that automate documentation search, and a Slack bot that investigates incidents using that search plus metric plugins and log APIs. Moving from script to service is mostly operations work:
- Evals. Keep 20 to 30 real questions, each with the runbook it should cite, and re-run them whenever chunking, prompt or model changes. Post-mortems are a rich source.
- Versioning. Log the model ID, a prompt hash and the index build time with every answer.
- Budgets. Track tokens per request, and set a latency SLO if the bot sits on the incident path, as in a day in the life of an AI-augmented SRE.
Common pitfalls: what breaks first
Most failures are retrieval failures that look like model failures. When an answer is wrong, print the retrieved chunks before you touch the prompt.
| Symptom | Likely cause | Fix |
|---|---|---|
| Misses the obvious runbook | Text past 256 tokens is never embedded | Split on headings, cap chunk size |
| Mixes steps from two services | Chunks lost their runbook title | Prefix every chunk with the title |
| Cites a retired command | Stale docs, or chunks of deleted files | Delete .chroma/ and rebuild in CI |
| Confident answer, weak support | Poor matches reached the model | Distance cut-off, refusal, citation check |
| Right runbook, wrong section | Vague headings or too few results | Task-style headings, more results |
Key takeaways
- A RAG pipeline has six stages: load, chunk, embed, store, retrieve, generate.
- Chunk runbooks on headings and keep the title in every chunk.
- Refuse when retrieval is weak, and make every answer cite its sources.
- Run it as a service: evals, pinned versions, token budgets.
Related articles

The Toil Audit: What SRE Work to Automate First
Stop guessing which toil to kill. Log two weeks of SRE toil, score each task by frequency, time, risk and stability, and climb the automation ladder from runbook to AI agent.

A Day in the Life of an AI-Augmented SRE (2026)
What actually changes when SREs work with AI every day: handoffs, triage, Terraform changes and reviews, block by block, with the commands that do the heavy lifting.