--- title: "Building AI Agents for DevOps Part 13: RAG Reading What Humans Wrote" url: "https://tirraflow.com/posts/building-ai-agents-for-devops-part-13-rag-reading-what-humans-wrote" author: "taoufiq" published: "2026-09-12T22:16:39Z" updated: "2026-10-22T09:39:01Z" category: "AI Agents" difficulty: "intermediate" series: "Building AI Agents for DevOps" series_part: 13 series_url: "https://tirraflow.com/series/building-ai-agents-for-devops" tags: ["Agents", "ADK"] --- # Building AI Agents for DevOps Part 13: RAG Reading What Humans Wrote > **Updated for Orrery 0.6.0 (October 2026).** Retrieval works as described. What changed around it: the set of specialists that can be asked is now discovered from installed packages, YAML and MCP servers, and admins can change it from the console without a restart. See [Part 17](https://tirraflow.com/posts/building-ai-agents-for-devops-part-17-opening-the-platform-without-opening-the-gates). In [Part 12](https://tirraflow.com/posts/building-ai-agents-for-devops-part-12-distributed-tracing-with-opentelemetry) I wired OpenTelemetry through the agent so a single incident turn became one trace you could replay. By then Orrery could read live infrastructure through about seventy tools, and remember its own past sessions. It still could not read a single thing a human had written. No runbook. No postmortem. No ADR. Every incident started from zero institutional knowledge — the agent rediscovering, from metrics and logs, something a colleague had written down eighteen months earlier and filed under _Kafka ISR shrink_. That is the gap this post closes, with one new tool called `search_knowledge`. What follows is mostly a list of decisions that look small and are not. --- ## Retrieval Is a Tool, Not Grounding This is the one I want to put first, because getting it wrong is silent. Google's ADK ships something called `VertexAiSearchTool`. The name says tool. It is not a tool. Its `process_llm_request` appends a `types.Retrieval` to the request config, and the _model_ does the retrieval server-side. That distinction matters because of what Orrery hangs off the tool path. Every tool result in the system passes through four plugins: - `SafetyScreenPlugin` — neutralizes prompt-injection spans inside the result - `PIIRedactionPlugin` — scrubs credentials - `ToolOutputCapPlugin` — bounds a chatty response - `AuditPlugin` — records what was asked Grounding has no `after_tool_callback`. It bypasses all four, and never appears in the audit log. Now ask the question that makes this urgent: **is a retrieved document trusted text?** It is not. A Confluence page is editable by anyone with write access to the space. A git-hosted runbook is editable by anyone who can open a PR. That is precisely the threat model of a pod annotation or a log line — text an attacker can reach, arriving inside a tool result, being read by a model that can restart things. So `search_knowledge` subclasses ADK's `BaseRetrievalTool` and travels the normal after-tool chain like every other tool. And because "do not wire the grounding tool" is the kind of rule that survives exactly as long as the person who wrote it remembers it, there is a test that walks both agent roots and fails the build if a grounding tool is ever attached. ```python def test_no_grounding_tool_attached(agent): """Grounding bypasses the after-tool chain: no screen, no audit.""" for tool in _walk_tools(agent): assert not isinstance(tool, VertexAiSearchTool), ( f"{tool!r} retrieves server-side; its result never reaches " "SafetyScreenPlugin, PIIRedactionPlugin, ToolOutputCapPlugin " "or AuditPlugin." ) ``` A managed backend is still reachable — through `DiscoveryEngineSearchTool`, which is a real `FunctionTool`, or through an adapter behind our own retriever protocol. The requirement is not "self-hosted". It is "on the tool path". --- ## Two Seams, Not One "RAG Provider" The obvious shape for this is a `RAGProvider` interface with `index()` and `search()`. I started there and it was wrong within a day. The reason is that the market splits down the middle of that interface. A managed service like Vertex AI Search owns _both_ halves — it ingests through its own connectors and answers queries, and you never write an indexer. A self-hosted store like Elasticsearch or pgvector owns _neither_ — it does nothing at all until you write both sides. One interface forces every managed backend to implement a fake `index()` that raises. So there are two protocols instead: | Seam | Shape | Who implements it | | --- | --- | --- | | `KnowledgeSource` | → `Document` | filesystem, git, Confluence | | `KnowledgeIndex` | write | only backends we populate | | `KnowledgeRetriever` | query → `Passage` | every backend | A backend that implements only `KnowledgeRetriever` is making a statement: _my ingestion is somebody else's job_. The sync command reads that and exits cleanly instead of failing. The payoff is in what a change costs. Adding Notion is one `KnowledgeSource`. Standardising on Azure AI Search is one `KnowledgeRetriever`. Neither touches an agent. --- ## Why the Vector Backend Is Hybrid Elasticsearch shipped first, BM25 only — no embedding provider, no cost, and `make up` already starts the container. Plain lexical search was a large improvement over what came before it, and it proved the seams before phase two paid for semantics. Then pgvector, and here is the thing I did not expect to have to argue. **Semantic search is weakest exactly where SRE queries are strongest.** The queries that matter in an incident are identifiers. `CrashLoopBackOff`. A consumer group name. An error string someone pasted from a log. You want the document containing that literal token — and a nearest-neighbour search will cheerfully hand back three plausible, well-written, semantically adjacent pages that never contain it at all. So pgvector computes _both_ rankings and fuses them with Reciprocal Rank Fusion: ```sql WITH semantic AS ( SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> CAST(:vec AS vector)) AS rank FROM orrery_knowledge_chunks WHERE embedding IS NOT NULL LIMIT 50 ), lexical AS ( SELECT id, ROW_NUMBER() OVER (ORDER BY ts_rank_cd(tsv, query) DESC) AS rank FROM orrery_knowledge_chunks, plainto_tsquery(:q) query WHERE tsv @@ query LIMIT 50 ) SELECT id, SUM(1.0 / (60 + rank)) AS score FROM (SELECT * FROM semantic UNION ALL SELECT * FROM lexical) fused GROUP BY id ORDER BY score DESC ``` RRF fuses **ranks, not scores**, and that is the whole point. Cosine distance and `ts_rank_cd` share no scale — one is bounded, the other is not — so any weighted sum of the raw numbers is dominated by whichever happens to have the larger range. You would be tuning a coefficient forever and calling it relevance. Two constants carry more thought than they look like they do. The candidate pool is 50 per ranking, wider than `top_k`, because a document ranked twentieth semantically and second lexically should still win — truncating each list at `top_k` would hide it before fusion ever saw it. And the constant 60 damps low-ranked hits, so a document has to place well in at least one ranking rather than accumulating credit for being mediocre in both. --- ## Chunking: Headings First, Then Size A fixed-width window is the default answer and it is bad for runbooks specifically. A runbook's "Recovery" section is a semantic unit. A 400-token window straddling the end of "Symptoms" and the start of "Recovery" retrieves half of each and reads as neither — the model gets a description of a problem welded to the middle of a fix. So chunking splits on headings first and only falls back to size within a section. That also hands every chunk a `section` for free, which is what makes a citation worth printing: _§ Recovery_ beats _chunk 47_. Two smaller decisions: - **Budgets are in characters, not tokens.** No tokenizer dependency, and chunking stays byte-identical across model providers. Roughly four characters per token for English prose is close enough for a budget. - **Fenced code blocks are never split.** A truncated YAML manifest or half a shell pipeline is worse than an oversized chunk, because the model will act on the fragment. An agent that can restart pods should never read half a command. --- ## Provenance Is Required, Not Optional Every passage carries where it came from: ```json { "text": "Restart the broker after confirming ISR has not recovered…", "source": "git://docs/runbooks/kafka-isr-shrink.md", "title": "Kafka ISR shrink", "section": "Recovery", "revision": "a1b2c3d", "age_days": 12, "stale": false, "score": 14.05 } ``` This is a required field on the model, not a nice-to-have that got added later. At 03:00 an operator cannot otherwise tell a retrieved fact from a hallucination — and the moment someone gets burned by that once, the team stops trusting retrieval entirely. Losing trust is the actual failure mode of RAG in operations, not bad recall. `stale` marks anything untouched for 180 days. It is deliberately a _flag_, not a filter: a two-year-old runbook may be the only one that exists, and the model should discount it rather than never see it. Related, and easy to get wrong: **"no matches" and "backend unreachable" are different results.** If they looked the same, a broken index would quietly answer "we have no runbook for that" — which is a sentence an operator will believe. That splits the failure policy in two. Misconfiguration fails fast at startup, because a pod that comes up healthy while serving an empty corpus is worse than one that refuses to boot. A backend that goes down at query time does _not_ fail the turn — knowledge is an augmentation, and the platform diagnosed incidents without it before this existed. --- ## Indexing Is a Build Step `make knowledge-sync` is explicit and runs in CI. It is never lazy at request time, because a first query that silently triggers a full Confluence crawl is an outage waiting for its moment. ```bash $ make knowledge-sync indexed 30 document(s) as 355 chunk(s); skipped 0 unchanged; pruned 0 stale chunk(s); removed 0 deleted document(s) ``` Sync is revision-driven, so a second pass costs one aggregation query and skips everything. Prefer git as the source over the filesystem: the revision is a commit sha, which is content-derived and stable across clones, so CI re-indexes only what a merge actually changed. Filesystem mtimes change on a fresh clone even when nothing did. Two behaviours worth stealing: - **Pruning after a shrinking edit.** A document that was nine chunks and is now four would otherwise keep answering from five slices that no longer exist anywhere. - **Pruning is skipped entirely when a run had any error.** A source that failed halfway looks identical to one whose documents were all deleted. Acting on that ambiguity empties your corpus because a Confluence token expired overnight. --- ## What It Deliberately Does Not Do Retrieval runs at `viewer` and is **not ACL-aware**. Rather than fake it, the rule is stated: index only what every viewer may read. The Confluence source enforces that at the edge — it refuses to auto-discover spaces. A constructor without an explicit space list raises, and a 403 fails loudly instead of being skipped. Auto-discovery would make restricted content readable through the agent with no trace on the Confluence side, which is a data leak wearing the costume of a convenience feature. Also not done, and worth naming: - **Retrieval-quality evals.** The existing 33 scenarios score tool trajectory, which is blind to whether the _right_ document came back. - **ACL-aware retrieval.** Deferred, on purpose — pretending to enforce permissions that are not enforced is worse than not indexing. --- ## Turning It On Off by default. Nothing changes until you set a backend. ```bash make up PROFILES=elastic export ORRERY_KNOWLEDGE_BACKEND=elasticsearch make knowledge-sync make run-api ``` Start with Elasticsearch. Move to pgvector when your queries start being phrased in the words of the incident rather than the words of the runbook — that is the moment lexical search stops being enough. Full configuration is in the [knowledge retrieval docs](https://bahalla.github.io/orrery/knowledge/), and the design record is [AEP-025](https://bahalla.github.io/orrery/enhancements/aep-025-knowledge-retrieval/). --- ## What's Next The agent now reads live infrastructure, its own history, and what the team wrote down. The obvious next gap is knowing whether retrieval is any _good_ — scoring the documents that come back, not just the tools that were called.