Building AI Agents for DevOps Part 13: RAG Reading What Humans Wrote

In Part 12 I wired OpenTelemetry through the agent so a single incident turn became one trace you could replay. By then Orrery could read live infrastructure through about seventy tools, and remember...

Updated for Orrery 0.6.0 (October 2026). Retrieval works as described. What changed around it: the set of specialists that can be asked is now discovered from installed packages, YAML and MCP servers, and admins can change it from the console without a restart. See Part 17.

In Part 12 I wired OpenTelemetry through the agent so a single incident turn became one trace you could replay. By then Orrery could read live infrastructure through about seventy tools, and remember its own past sessions.

It still could not read a single thing a human had written.

No runbook. No postmortem. No ADR. Every incident started from zero institutional knowledge — the agent rediscovering, from metrics and logs, something a colleague had written down eighteen months earlier and filed under Kafka ISR shrink. That is the gap this post closes, with one new tool called search_knowledge.

What follows is mostly a list of decisions that look small and are not.

Retrieval Is a Tool, Not Grounding

This is the one I want to put first, because getting it wrong is silent.

Google's ADK ships something called VertexAiSearchTool. The name says tool. It is not a tool. Its process_llm_request appends a types.Retrieval to the request config, and the model does the retrieval server-side.

That distinction matters because of what Orrery hangs off the tool path. Every tool result in the system passes through four plugins:

Grounding has no after_tool_callback. It bypasses all four, and never appears in the audit log.

Now ask the question that makes this urgent: is a retrieved document trusted text?

It is not. A Confluence page is editable by anyone with write access to the space. A git-hosted runbook is editable by anyone who can open a PR. That is precisely the threat model of a pod annotation or a log line — text an attacker can reach, arriving inside a tool result, being read by a model that can restart things.

So search_knowledge subclasses ADK's BaseRetrievalTool and travels the normal after-tool chain like every other tool. And because "do not wire the grounding tool" is the kind of rule that survives exactly as long as the person who wrote it remembers it, there is a test that walks both agent roots and fails the build if a grounding tool is ever attached.

def test_no_grounding_tool_attached(agent):
    """Grounding bypasses the after-tool chain: no screen, no audit."""
    for tool in _walk_tools(agent):
        assert not isinstance(tool, VertexAiSearchTool), (
            f"{tool!r} retrieves server-side; its result never reaches "
            "SafetyScreenPlugin, PIIRedactionPlugin, ToolOutputCapPlugin "
            "or AuditPlugin."
        )

A managed backend is still reachable — through DiscoveryEngineSearchTool, which is a real FunctionTool, or through an adapter behind our own retriever protocol. The requirement is not "self-hosted". It is "on the tool path".

Two Seams, Not One "RAG Provider"

The obvious shape for this is a RAGProvider interface with index() and search(). I started there and it was wrong within a day.

The reason is that the market splits down the middle of that interface. A managed service like Vertex AI Search owns both halves — it ingests through its own connectors and answers queries, and you never write an indexer. A self-hosted store like Elasticsearch or pgvector owns neither — it does nothing at all until you write both sides.

One interface forces every managed backend to implement a fake index() that raises.

So there are two protocols instead:

Seam

Shape

Who implements it

KnowledgeSource

→ Document

filesystem, git, Confluence

KnowledgeIndex

write

only backends we populate

KnowledgeRetriever

query → Passage

every backend

A backend that implements only KnowledgeRetriever is making a statement: my ingestion is somebody else's job. The sync command reads that and exits cleanly instead of failing.

The payoff is in what a change costs. Adding Notion is one KnowledgeSource. Standardising on Azure AI Search is one KnowledgeRetriever. Neither touches an agent.

Why the Vector Backend Is Hybrid

Elasticsearch shipped first, BM25 only — no embedding provider, no cost, and make up already starts the container. Plain lexical search was a large improvement over what came before it, and it proved the seams before phase two paid for semantics.

Then pgvector, and here is the thing I did not expect to have to argue.

Semantic search is weakest exactly where SRE queries are strongest.

The queries that matter in an incident are identifiers. CrashLoopBackOff. A consumer group name. An error string someone pasted from a log. You want the document containing that literal token — and a nearest-neighbour search will cheerfully hand back three plausible, well-written, semantically adjacent pages that never contain it at all.

So pgvector computes both rankings and fuses them with Reciprocal Rank Fusion:

WITH semantic AS (
    SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> CAST(:vec AS vector)) AS rank
    FROM orrery_knowledge_chunks
    WHERE embedding IS NOT NULL
    LIMIT 50
), lexical AS (
    SELECT id, ROW_NUMBER() OVER (ORDER BY ts_rank_cd(tsv, query) DESC) AS rank
    FROM orrery_knowledge_chunks, plainto_tsquery(:q) query
    WHERE tsv @@ query
    LIMIT 50
)
SELECT id, SUM(1.0 / (60 + rank)) AS score
FROM (SELECT * FROM semantic UNION ALL SELECT * FROM lexical) fused
GROUP BY id ORDER BY score DESC

RRF fuses ranks, not scores, and that is the whole point. Cosine distance and ts_rank_cd share no scale — one is bounded, the other is not — so any weighted sum of the raw numbers is dominated by whichever happens to have the larger range. You would be tuning a coefficient forever and calling it relevance.

Two constants carry more thought than they look like they do. The candidate pool is 50 per ranking, wider than top_k, because a document ranked twentieth semantically and second lexically should still win — truncating each list at top_k would hide it before fusion ever saw it. And the constant 60 damps low-ranked hits, so a document has to place well in at least one ranking rather than accumulating credit for being mediocre in both.

Chunking: Headings First, Then Size

A fixed-width window is the default answer and it is bad for runbooks specifically.

A runbook's "Recovery" section is a semantic unit. A 400-token window straddling the end of "Symptoms" and the start of "Recovery" retrieves half of each and reads as neither — the model gets a description of a problem welded to the middle of a fix.

So chunking splits on headings first and only falls back to size within a section. That also hands every chunk a section for free, which is what makes a citation worth printing: § Recovery beats chunk 47.

Two smaller decisions:

Provenance Is Required, Not Optional

Every passage carries where it came from:

{
  "text": "Restart the broker after confirming ISR has not recovered…",
  "source": "git://docs/runbooks/kafka-isr-shrink.md",
  "title": "Kafka ISR shrink",
  "section": "Recovery",
  "revision": "a1b2c3d",
  "age_days": 12,
  "stale": false,
  "score": 14.05
}

This is a required field on the model, not a nice-to-have that got added later. At 03:00 an operator cannot otherwise tell a retrieved fact from a hallucination — and the moment someone gets burned by that once, the team stops trusting retrieval entirely. Losing trust is the actual failure mode of RAG in operations, not bad recall.

stale marks anything untouched for 180 days. It is deliberately a flag, not a filter: a two-year-old runbook may be the only one that exists, and the model should discount it rather than never see it.

Related, and easy to get wrong: "no matches" and "backend unreachable" are different results. If they looked the same, a broken index would quietly answer "we have no runbook for that" — which is a sentence an operator will believe.

That splits the failure policy in two. Misconfiguration fails fast at startup, because a pod that comes up healthy while serving an empty corpus is worse than one that refuses to boot. A backend that goes down at query time does not fail the turn — knowledge is an augmentation, and the platform diagnosed incidents without it before this existed.

Indexing Is a Build Step

make knowledge-sync is explicit and runs in CI. It is never lazy at request time, because a first query that silently triggers a full Confluence crawl is an outage waiting for its moment.

$ make knowledge-sync
indexed 30 document(s) as 355 chunk(s); skipped 0 unchanged; pruned 0 stale chunk(s); removed 0 deleted document(s)

Sync is revision-driven, so a second pass costs one aggregation query and skips everything. Prefer git as the source over the filesystem: the revision is a commit sha, which is content-derived and stable across clones, so CI re-indexes only what a merge actually changed. Filesystem mtimes change on a fresh clone even when nothing did.

Two behaviours worth stealing:

What It Deliberately Does Not Do

Retrieval runs at viewer and is not ACL-aware. Rather than fake it, the rule is stated: index only what every viewer may read.

The Confluence source enforces that at the edge — it refuses to auto-discover spaces. A constructor without an explicit space list raises, and a 403 fails loudly instead of being skipped. Auto-discovery would make restricted content readable through the agent with no trace on the Confluence side, which is a data leak wearing the costume of a convenience feature.

Also not done, and worth naming:

Turning It On

Off by default. Nothing changes until you set a backend.

make up PROFILES=elastic
export ORRERY_KNOWLEDGE_BACKEND=elasticsearch
make knowledge-sync
make run-api

Start with Elasticsearch. Move to pgvector when your queries start being phrased in the words of the incident rather than the words of the runbook — that is the moment lexical search stops being enough.

Full configuration is in the knowledge retrieval docs, and the design record is AEP-025.

What's Next

The agent now reads live infrastructure, its own history, and what the team wrote down. The obvious next gap is knowing whether retrieval is any good — scoring the documents that come back, not just the tools that were called.

Read on Tirraflow