Building AI Agents for DevOps Part 17: Opening the Platform Without Opening the Gates

Until Orrery 0.6.0, I had written every tool the agent could call. Each one was a Python function in this repository, reviewed and labelled with @confirm or @destructive when it changed something....

Until Orrery 0.6.0, I had written every tool the agent could call. Each one was a Python function in this repository, reviewed and labelled with @confirm or @destructive when it changed something. Every gate described in this series reads those labels: RBAC, the autonomy levels and the approval card.

0.6.0 opens the platform in four places:

• specialists from other Python packages;

• specialists written in YAML;

• tools from MCP servers;

• admins changing the set of agents from the console, while it runs.

It also lets calls reach Kubernetes and Elasticsearch as the person who asked. Each of these is a way for something I never reviewed to reach the gates. This post covers how each one fails closed, and the bug that showed it didn't, at first.

A Tool With No Function

The gates found a tool's guard level on its Python function. An MCP tool has no Python function: it is discovered at runtime from a server. Writing the MCP integration made me read every gate's first lines, and all four began the same way:

def callback(*, tool, args, tool_context):
    func = getattr(tool, "func", None)
    if func is None:
        return None          # no function: let it through

No shipped tool was affected, because every one of them was a function. But every MCP tool would have taken that early return and run without a gate, whatever it did. A viewer could have called a server's delete_repository and never seen an approval card. Nothing failed, because no test had ever exercised a tool that wasn't a function.

The fix has three parts:

• Read the label from the tool itself when there is no function: func = getattr(tool, "func", tool). A new set_guard() labels tool objects.

• create_agent() attaches the approval gate itself to any agent holding a guarded tool or a toolset, whose tools are only known at runtime. Wiring the gate by hand had already been forgotten twice: once on the Docker specialist, once on the remediation actor.

• The wiring test asks for a real gate. It used to accept any before-tool callback, so a metrics callback in that slot passed while gating nothing. Gates are now marked with mark_confirmation_gate(), and the test requires the mark.

The MCP tests start a real stdio MCP server and prove that a destructive call never reaches it without approval.

MCP: Fail Closed on Someone Else's Tools

Each configured MCP server becomes a specialist on the chat root. Every tool it exposes gets a guard level as it is discovered, with one rule above the others: when no rule says otherwise, the tool is destructive.

servers:
  - name: github                      # tools appear as github_*
    description: GitHub issues and pull requests for the platform repos
    transport: streamable_http
    url: https://api.githubcopilot.com/mcp/
    headers_from_env:                 # never inline a secret
      Authorization: GITHUB_MCP_AUTHORIZATION
    tools: [list_issues, get_issue, create_issue]   # required allowlist
    read_only: [list_issues, get_issue]
    # create_issue is not classified, so it is destructive

The rule that took the most thought is about MCP annotations. A server can describe its tools as readOnlyHint or destructiveHint, and it is tempting to believe it. But an annotation is the server describing itself. So Orrery trusts annotations in one direction only: a hint that raises protection is always applied, and a hint that lowers it is ignored unless the operator sets trust_annotations. A reviewed classification in the config always wins. If you classify a tool, it can only get cheaper to call, never more dangerous.

The transport is locked down too. HTTP servers must use https unless explicitly allowed. Header secrets come from environment variables, never the file. Stdio servers start with an empty environment, so a local MCP process doesn't inherit the agent's cloud credentials. The Helm chart can render the config and refuses to enable MCP with no servers.

Specialists From Anywhere

The chat root used to import each specialist by name, so adding one meant editing the orchestrator. Specialists now register through a Python entry point, and the root discovers what is installed:

[project.entry-points."orrery.specialists"]
k8s = "k8s_health_agent.agent:root_agent"

A specialist from another package needs no change to Orrery. ORRERY_SPECIALISTS and ORRERY_DISABLED_SPECIALISTS choose what runs, and an unknown name stops startup instead of being skipped quietly.

Often a new specialist is just a narrower brief over tools that already exist. For that, you don't need a package. A declarative agent is a few lines of YAML:

agents:
  - name: first_responder
    description: First look at a paging alert, before anyone changes anything
    instruction: Check events, then the alerts, then the affected workload.
    tools: [k8s.get_events, get_active_alerts]

The important limit is that a declarative agent can only recombine reviewed code. Its tools come from the catalogue of installed specialists, each keeps its own guard level, and the gate is attached automatically. An unknown or ambiguous tool name, or a name collision, stops startup. The console's new Agents tab lists every specialist, where it came from (built-in, plugin, YAML or MCP), and the guard level of each tool.

Editing Agents in a Running System

With ORRERY_ADMIN_EDITING=true, admins can do the same from that tab: switch specialists off, and compose, edit or remove declarative agents, with a tool picker that shows guard levels. The change applies without a restart.

The whole design rests on one invariant: a replica only ever serves an agent tree that was built successfully. So the order is build, then save, then swap:

• Build first. Building the new tree is the validation. If it fails, the request gets a 422 and nothing is stored.

• Save with a version check. Every save names the version it was edited from. If another admin saved in between, the request gets a 409 instead of silently overwriting their change. Every save is audited.

• Swap the root. Turns already running finish on the agents they started with, and the next turn gets the new set. The deployment's environment stays the ceiling: an admin can't enable a specialist the environment excludes.

Other replicas poll the store and follow within ORRERY_CATALOG_POLL_SECONDS. The review before the release found the weak spot in that loop: one unexpected error would end the polling task for good, and that replica would keep serving a stale catalog, silently, until its next restart. A background loop that can die needs a test that makes it fail once and checks it is still polling afterwards. It has one now.

Acting as the User

Orrery's RBAC decides whether a tool may run. With ORRERY_IMPERSONATION=kubernetes,elasticsearch, the call itself also reaches the target system as the person who asked: Kubernetes receives Impersonate-User and Impersonate-Group, and Elasticsearch receives es-security-runas-user. The cluster's own permissions and audit log then apply to that person, on top of Orrery's.

The identity comes only from what a transport verified: the JWT in the console, the signed sender in Google Chat, or the Slack user's email from users.info. It is kept in a ContextVar for the duration of the turn, never in session state, because tools can write session state. Three rules keep it fail-closed:

• No verified user, no call. A call with nobody behind it is refused, not quietly sent as the service account. The unattended triage sweep declares that it acts as the platform, in code (service_scope()). Falling back to the service account is an explicit rollout option.

• Header values are validated. A claim containing CR/LF could otherwise add headers of its own.

• An email must be verified. This one was also found in the pre-release review. The default user claim was email, and a valid signature proves the identity provider issued the token, not that the address belongs to that person. Where users can edit their own email, someone could set a colleague's address and be impersonated as that colleague. The email now counts only with email_verified: true. Otherwise Orrery uses the subject, which Kubernetes doesn't know, so the target denies the call instead of running it as the wrong person.

A mistyped ORRERY_IMPERSONATION value now stops startup, instead of failing on the first Kubernetes call. Kafka and MCP servers use tokens rather than impersonation headers. ADR-004 proposes token exchange for them, and until then they run as the service account.

Hardening the Pipeline That Ships It

All of this runs through a CI pipeline whose tokens can push images, sign them and publish releases, so it got the same review:

• Every third-party action is pinned to a commit SHA. Tags can be moved to point at different code. Dependabot keeps the pins current in one grouped weekly PR.

• Least-privilege tokens. Workflows default to contents: read, and each release job asks only for the permissions it uses.

• A script injection, fixed. The Dependabot relock workflow put the PR title straight into a shell script with ${{ }}. A title containing shell syntax would have run with a token that can push to the repository. It now goes through an environment variable.

• A new Workflow Lint job runs actionlint and the zizmor security audit on every PR, so these mistakes fail the build.

The Takeaway

Every extension point is a place where something unreviewed enters a reviewed system. The question for each one is the same: when nobody has looked at it, what does it default to? The answer should always be "the most protected thing": a destructive tool, a refused call, the last agent tree that built. And the bug at the start is why every default needs a test that actually goes through it. That early return None had been in the gates for months. It was harmless only because nothing had ever reached it.

The full list is in the v0.6.0 release notes. The MCP setup is documented in the MCP guide, and impersonation in the impersonation guide.