Building AI Agents for DevOps Part 10: Fixing the Approve Handshake Across Sub-Agents

In Part 9 I shipped the Google Chat transport for the Orrery DevOps assistant — same root agent, same RBAC, same guardrails, just a different front door. @destructive tools posted a Card v2, the...

Updated for Orrery 0.6.0 (October 2026). This handshake became stricter. Every shipped surface now requires a deliberate word (approve, not ok) from the same verified person who triggered the action, given after the action existed. The keys it relies on are protected by an identity guard that reverts any change a tool makes to them (Part 15).

In Part 9 I shipped the Google Chat transport for the Orrery DevOps assistant — same root agent, same RBAC, same guardrails, just a different front door. @destructive tools posted a Card v2, the operator clicked Approve, the agent re-ran the tool, life was good.

Then a real conversation happened.

You      can you patch deployment in gatekeeper namespace ?
bot      Of course. What is the deployment and the patch?
You      patch memory request and limit on gatekeeper-audit to 1Gi
bot      ⚠️ DESTRUCTIVE: patch_deployment   [Approve] [Deny]
You      /Approve
bot      Approved by Taoufiq — executing patch_deployment
bot      The remediation attempt revealed a deeper issue: the
         frontend deployment is entirely missing, so the service
         is down…

The Incident: Hallucination on Approve

The patch never ran. This is a story about how tool_context.session.id silently broke the approve handshake at the AgentTool boundary.

The Smoking Gun in the Audit Log

// Turn 1: Sub-agent request
{"tool": "k8s_health_agent.patch_deployment", "session_id": "60b04298-..."}
{"tool": "orrery_assistant.k8s_health_agent", "session_id": "gchat:spaces/..."}

// Turn 2: User clicks Approve
{"event": "Starting agent run", "session_id": "60b04298-..."}

The bot re-entered on the Inner Session ID — a fresh, empty conversation with no history. Without the context of the Gatekeeper request, the LLM guessed and hallucinated.

The Bug: Session ID Drift

The Google Chat before_tool_callback ran on every guarded tool. When it fired, it stashed the session_id to allow the click handler to resume execution on Approve.

For tools inside an AgentTool — like k8s_health_agent.patch_deployment — the ADK spawns an inner session. The store ended up holding this ephemeral UUID, but the click handler (triggered by the user in the main thread) was looking for the parent session. The handshake was broken for every specialist agent.

Why Slack Worked (By Accident)

The Slack bot only wired the callback at the Root Agent. Sub-agent tools fell back to text-based require_confirmation(). The Google Chat bot, seeking a better UX, wired the callback into every sub-agent, inadvertently exposing this architectural mismatch.

Integrating Google Chat Commands

To make the bot robust, I integrated Google Chat Commands. Instead of just buttons, it now supports Slash and Quick Commands like /Approve and /Deny.

1. Configuration

I defined these commands in the Google Chat API configuration:

2. Handling Interaction Events

When a user types /Approve, Google Chat sends a MESSAGE event containing the slashCommand object. The handler maps the commandId to the internal approval logic:

async def on_message(event):
    if "slashCommand" in event:
        command_id = event["slashCommand"]["commandId"]
        if command_id == "1":  # Approve
            return await handle_approval(event)
        elif command_id == "2":  # Deny
            return await handle_denial(event)

The Cascade: Why State Propagation Fails

Even with the correct session ID, the handshake would fail because the original callback relied on tool_context.state to flag retries. However, AgentTool sessions are ephemeral:

1. Turn 1: Parent calls Sub-agent. Sub-agent writes pending=True to its inner state.

2. Turn 1 Ends: Inner session is destroyed. State is lost.

3. Approve Click: Parent runner starts.

4. Turn 2: Parent calls Sub-agent again. Sub-agent starts with empty state, sees no pending=True, and blocks again.

Result: The bot enters a "Loop of Doom," repeatedly asking for approval because it has no memory of the previous turn's guarded state.

Redesign: The Store Becomes the Source of Truth

The fix moves the entire approve handshake out of ephemeral session state and onto a persistent ConfirmationStore. This ensures the "Truth" survives the destruction of sub-agent sessions.

Implementation Details

1. Resolving the Parent Session

def _resolve_parent_session_id(tool_context: Context) -> str:
    state = getattr(tool_context, "state", None) or {}
    thread = state.get("gchat_thread")
    space = state.get("gchat_space", "")
    return f"gchat:{thread or space}" if (thread or space) else "unknown"

2. One-Shot Consumption

def consume_approved(self, thread_key, tool_name, args_hash):
    # Matches thread, tool, AND arguments within a 120s window
    # One-shot: pop() ensures it can't be reused
    return self._pending.pop(action_id) if valid else None

The Secure Re-entry Sequence

When an operator sends /Approve, the bot doesn't just "hit enter." It reconstructs the entire context to prevent hallucination through a synthetic prompt.

The LLM no longer reconstructs anything. It re-issues the exact call shown on the card.

One UX Polish: A Progress Card On Approve

While I was rewriting the click path, I noticed it had a small papercut: after Approve, the user stared at a silent thread for ~30 seconds while the LLM thought, then a wall of text appeared. The MESSAGE path already posts a live "🔍 Investigating…" card and PATCHes it as the run progresses. The Approve path didn't.

One copy-paste later it does:

progress_message_name = await self._post_initial_progress(
    space_name=pending.space_name, thread_name=pending.thread_name
)
tracker = self._make_tracker(progress_message_name)

result = await self._run_agent(
    session_id=pending.session_id,   # ← now the gchat parent session
    user_id=pending.user_id,
    user_text=synthetic_text,         # ← now embeds args
    user_role=self.resolve_role(pending.user_id),
    space_name=pending.space_name,
    thread_name=pending.thread_name,
    tracker=tracker,
)

await self._update_or_post(
    space_name=pending.space_name,
    thread_name=pending.thread_name,
    message_name=progress_message_name,
    reply={"text": combined_text, "cardsV2": result.get("cardsV2")},
)

The user gets the same in-place updating card during Approve as during a normal triage. Same code, same tracker, same debounce.

What "Working" Looks Like Now

End-to-end test, real GKE cluster, real OOMKilled gatekeeper-audit pod:

You      can you patch deployment in gatekeeper namespace ?
bot      Of course. Which deployment and what patch?
You      list the deployments in gatekeeper-system
bot      [table of gatekeeper-audit (0/1) and gatekeeper-controller-manager (1/1)]
You      can you restart gatekeeper-audit in gatekeeper-system
bot      ⚠️ DESTRUCTIVE: restart_deployment    Send Approve or Deny
You      /Approve
bot      Approved by Taoufiq — executing restart_deployment
bot      Deployment 'gatekeeper-audit' has been restarted.
You      patch memory request and limit on gatekeeper-audit to 1Gi
bot      [describes pod, finds OOMKilled, container=manager]
bot      ⚠️ DESTRUCTIVE: patch_deployment
         namespace=gatekeeper-system, name=gatekeeper-audit,
         patch={spec.template.spec.containers[manager].resources …1Gi}
         Send Approve or Deny
You      /Approve
bot      Approved by Taoufiq — executing patch_deployment
bot      Patch of deployment 'gatekeeper-audit' was successful.

Two destructive flows in the same conversation. Both Approves landed on the right tool with the right args. No detours through remediation_pipeline. No hallucinated frontend.

Lessons Learned

A Note on Production Deployments: The Multi-Replica Problem

The code snippets in this article show an in-memory ConfirmationStore using a Python dictionary and a threading lock. This is great for a local demo, but it carries a hidden trap for production.

As noted in the Orrery Deployment Documentation, the project is designed for Kubernetes with multiple replicas. In a multi-pod environment, an in-memory store is a dealbreaker:

1. Turn 1: User clicks /Approve. The request hits Pod A. Pod A marks the action as approved in its local memory.

2. Turn 2: The LLM Runner re-enters the agent. Kubernetes load-balances the request to Pod B.

3. Failure: Pod B checks its own local memory, finds nothing, and issues a fresh approval card. The user is stuck in the "Loop of Doom" again.

The Fix: For production, the ConfirmationStore must be backed by a distributed database (like Postgres) or a cache (like Redis). By moving the handshake to the same database that stores your session history, you ensure that any pod in your cluster can validate an approval, regardless of where the original click landed.

What Changed (For Reference)

Try It Yourself

The bug-and-fix lives in agents/google-chat-bot/google_chat_bot/confirmation.py. For more detailed guides and architectural overview, check out the Orrery Project Documentation.

Two short reads explain everything:

1. _resolve_parent_session_id — why state, not tool_context.session.id.

2. ConfirmationStore.consume_approved — the single primitive that gates retry through three predicates and a time window.

To reproduce locally, run the bot against a cluster that has any @destructive Kubernetes tool registered, ask it to patch something, click Approve, and watch the audit log. Before this fix, you'd see two different session_id values bracketing the click. After, you'd see one — the gchat one.

What's Next

The same "don't trust the inner session" pattern likely affects any callback we wire across an AgentTool tree — RBAC re-evaluation, metric tagging, audit correlation. The Google Chat bug was the first place it bit, but it won't be the last. Next stop: a small audit of every before_tool_callback we register and an explicit test for the AgentTool boundary in each one.

The full source code is on GitHub. Star it if you find it useful.