RAG 101: Build Your First RAG Pipeline over Runbooks

11 min readBy Abhishek Kothari#AI for SRE
RAG 101: Build Your First RAG Pipeline over Runbooks

Every team has runbooks, and at 3 a.m. nobody can find the right one. A RAG pipeline (retrieval-augmented generation) fixes that: it searches your docs first, then lets an LLM answer only from what it found. In this guide you will build one over a folder of Markdown runbooks in about 50 lines of Python, get answers that cite their sources, and learn what breaks first.

How a RAG pipeline works

A large language model (LLM) only knows its training data, so it has never read your checkout runbook. RAG fixes that at question time: find the passages that match the question, paste them into the prompt, and tell the model to answer from them alone. The diagram shows the six stages on two paths.

Architecture of a RAG pipeline: the index path loads runbooks, chunks them, embeds the chunks locally and stores them in Chroma; the query path retrieves the closest chunks for an SRE's question and Claude generates a cited answer
Index when runbooks change; retrieve and generate on every question.

The top row runs whenever a runbook changes. It loads each file, chunks it into sections, embeds each chunk and stores the result. An embedding is a list of numbers that places text with similar meaning close together, so you can search by meaning instead of keywords. The bottom row runs per question: it retrieves the nearest chunks and generates an answer that cites them.

You need Python 3.9 or newer, an Anthropic API key and a few Markdown runbooks. Chroma is the vector database: it runs inside your Python process, saves to a local folder and ships a default embedding model (all-MiniLM-L6-v2) that runs on your CPU. Only the question and the matching excerpts leave your machine, sent to Claude through the Anthropic Python SDK.

Step 1: Set up the project

Start with a clean virtual environment. Chroma pulls in ONNX Runtime to run the embedding model locally, and the Anthropic SDK reads your key from an environment variable, so it never lands in code.

python3 -m venv .venv && source .venv/bin/activate
pip install chromadb anthropic
export ANTHROPIC_API_KEY="your-key-here"
ls runbooks/
# cert-rotation.md  checkout-5xx.md  disk-full.md  kafka-consumer-lag.md

Four or five real runbooks are plenty for a first test. If import chromadb later fails with "Chroma requires sqlite3 >= 3.35.0", your Python was built against an old system SQLite. A newer Python from your package manager, pyenv or Homebrew fixes it.

Step 2: Load, chunk and index the runbooks

Chunking decides what the model gets to read, so it matters more than any other setting. Runbooks already have one heading per step, and splitting on those headings keeps each command next to its explanation. The diagram shows one runbook becoming four chunks.

The runbook checkout-5xx.md is split on its headings into four chunks; each chunk keeps the runbook title, gets an ID such as checkout-5xx.md#2 and an embedding vector
One chunk per section, each carrying its title, ID and source.

Each chunk starts with the runbook title, so a section called "Roll back" still says which service it belongs to. Its ID and source path are what answers will cite later. Save this as ingest.py:

import pathlib, re
import chromadb

MAX_CHARS = 1000  # the embedder reads only 256 tokens
col = chromadb.PersistentClient(path=".chroma").get_or_create_collection(
    name="runbooks", configuration={"hnsw": {"space": "cosine"}})

def chunks(text):
    # split on ## and ### headings, not on "# " comments in code
    for section in re.split(r"\n(?=#{2,3} )", text):
        for i in range(0, len(section), MAX_CHARS):
            if piece := section[i:i + MAX_CHARS].strip():
                yield piece

for path in sorted(pathlib.Path("runbooks").glob("**/*.md")):
    text = path.read_text(encoding="utf-8")
    title = text.splitlines()[0].lstrip("# ")
    parts = list(chunks(text))
    col.delete(where={"source": str(path)})  # drop stale chunks
    col.upsert(
        ids=[f"{path.name}#{n}" for n in range(len(parts))],
        documents=[f"{title}\n{p}" for p in parts],
        metadatas=[{"source": str(path), "chunk": n} for n in range(len(parts))])
    print(f"{path}: {len(parts)} chunks")
print("total chunks:", col.count())

PersistentClient keeps the index in .chroma/, and the cosine setting makes distances easy to read: 0 means the same meaning, larger means further apart. The delete before the upsert matters: without it, a runbook that shrinks from six sections to four keeps two stale chunks. The first run downloads the embedding model into ~/.cache/chroma, then prints a line per file:

$ python ingest.py
runbooks/cert-rotation.md: 3 chunks
runbooks/checkout-5xx.md: 4 chunks
runbooks/disk-full.md: 3 chunks
runbooks/kafka-consumer-lag.md: 5 chunks
total chunks: 15

A file that reports one chunk has no level-two headings, so it was cut blindly every 1,000 characters. Add headings rather than tuning the splitter, and re-run the script from CI whenever the runbooks change.

Step 3: Retrieve and generate a cited answer

Now the query path. The flowchart below shows its two decisions. The first matters most: if nothing in the index is close to the question, the script refuses instead of letting the model improvise.

Flowchart: an SRE asks a question, the script retrieves the top four chunks and checks whether any is close enough; if not it refuses and the team writes the missing runbook; if so Claude drafts a cited answer, and the SRE checks the citations before acting or fixing the chunk and re-indexing
Cyan steps are automated; green steps belong to the on-call engineer.

The model never acts on its own. It drafts, and a human checks the citations before running anything. Both "no" branches end in a docs fix, so every failure improves the runbooks. Save this as ask.py:

import os, sys
import anthropic, chromadb

MODEL = os.environ.get("RAG_MODEL", "claude-sonnet-5-5")  # pin and version it
MAX_DISTANCE = 0.6  # tune on your own questions
question = " ".join(sys.argv[1:])

col = chromadb.PersistentClient(path=".chroma").get_collection(name="runbooks")
hits = col.query(query_texts=[question], n_results=4)  # one result list per query
context = [(f"{m['source']}#{m['chunk']}", doc) for doc, m, dist in
           zip(hits["documents"][0], hits["metadatas"][0], hits["distances"][0])
           if dist <= MAX_DISTANCE]
if not context:
    sys.exit("No runbook covers this. Escalate to a human.")

excerpts = "\n\n".join(f"[{i}] {src}\n{doc}" for i, (src, doc) in enumerate(context, 1))
reply = anthropic.Anthropic().messages.create(  # reads ANTHROPIC_API_KEY
    model=MODEL, max_tokens=500,
    system="Answer only from the numbered runbook excerpts. Cite them like [1]. "
           "If they do not contain the answer, say so.",
    messages=[{"role": "user", "content": f"{excerpts}\n\nQuestion: {question}"}])
print(reply.content[0].text, "\n\nSources:")
for i, (src, _) in enumerate(context, 1):
    print(f"  [{i}] {src}")
print(f"tokens in/out: {reply.usage.input_tokens}/{reply.usage.output_tokens}")

n_results=4 fetches the four nearest chunks, and the distance cut-off drops weak matches. If none survive, sys.exit prints the refusal and exits with status 1 before a single token is spent. The system prompt limits Claude to the numbered excerpts and asks for citations. RAG_MODEL lets you swap the model without a code change, and the last line prints token usage: your cost per request.

Run it: ask a question, check the sources

Ask one real on-call question and one the runbooks cannot answer. The output is illustrative; yours depends on your runbooks.

$ python ask.py "checkout returns 5xx after a deploy, what first?"
Check the latest rollout with kubectl rollout history
deployment/checkout -n shop [1]. If a revision landed just
before the errors, roll back with kubectl rollout undo [2].

Sources:
  [1] runbooks/checkout-5xx.md#1
  [2] runbooks/checkout-5xx.md#2
tokens in/out: 912/74

$ python ask.py "how do I reset the payroll admin password?"
No runbook covers this. Escalate to a human.

Check two things in every answer: each claim carries a citation, and each cited chunk really says what the answer claims. The refusal is a feature, because it names a runbook you are missing. If good questions get refused, raise the cut-off slightly; if off-topic questions get answers, lower it.

From script to service: evals, versions and budgets

I have built RAG pipelines that automate documentation search, and a Slack bot that investigates incidents using that search plus metric plugins and log APIs. Moving from script to service is mostly operations work:

  • Evals. Keep 20 to 30 real questions, each with the runbook it should cite, and re-run them whenever chunking, prompt or model changes. Post-mortems are a rich source.
  • Versioning. Log the model ID, a prompt hash and the index build time with every answer.
  • Budgets. Track tokens per request, and set a latency SLO if the bot sits on the incident path, as in a day in the life of an AI-augmented SRE.

Common pitfalls: what breaks first

Most failures are retrieval failures that look like model failures. When an answer is wrong, print the retrieved chunks before you touch the prompt.

SymptomLikely causeFix
Misses the obvious runbookText past 256 tokens is never embeddedSplit on headings, cap chunk size
Mixes steps from two servicesChunks lost their runbook titlePrefix every chunk with the title
Cites a retired commandStale docs, or chunks of deleted filesDelete .chroma/ and rebuild in CI
Confident answer, weak supportPoor matches reached the modelDistance cut-off, refusal, citation check
Right runbook, wrong sectionVague headings or too few resultsTask-style headings, more results

Key takeaways

  • A RAG pipeline has six stages: load, chunk, embed, store, retrieve, generate.
  • Chunk runbooks on headings and keep the title in every chunk.
  • Refuse when retrieval is weak, and make every answer cite its sources.
  • Run it as a service: evals, pinned versions, token budgets.

Related articles