---
title: "Building AI Agents for DevOps Part 10: Fixing the Approve Handshake Across Sub-Agents"
url: "https://tirraflow.com/posts/building-ai-agents-for-devops-part-10-fixing-the-approve-handshake-across-sub"
author: "taoufiq"
published: "2026-04-26T12:33:54Z"
updated: "2026-10-15T21:16:04Z"
category: "Security"
difficulty: "advanced"
series: "Building AI Agents for DevOps"
series_part: 10
series_url: "https://tirraflow.com/series/building-ai-agents-for-devops"
tags: ["Agents", "ADK", "Security"]
---

# Building AI Agents for DevOps Part 10: Fixing the Approve Handshake Across Sub-Agents

> **Updated for Orrery 0.6.0 (October 2026).** This handshake became stricter. Every shipped surface now requires a deliberate word (_approve_, not _ok_) from the same verified person who triggered the action, given after the action existed. The keys it relies on are protected by an identity guard that reverts any change a tool makes to them ([Part 15](https://tirraflow.com/posts/building-ai-agents-for-devops-part-15-security-conventions-into-mechanisms)).

In [Part 9](https://tirraflow.com/posts/building-ai-agents-for-devops-part-9-google-chat-integration) I shipped the Google Chat transport for the Orrery DevOps assistant — same root agent, same RBAC, same guardrails, just a different front door. `@destructive` tools posted a Card v2, the operator clicked Approve, the agent re-ran the tool, life was good.

Then a real conversation happened.

```
You      can you patch deployment in gatekeeper namespace ?
bot      Of course. What is the deployment and the patch?
You      patch memory request and limit on gatekeeper-audit to 1Gi
bot      ⚠️ DESTRUCTIVE: patch_deployment   [Approve] [Deny]
You      /Approve
bot      Approved by Taoufiq — executing patch_deployment
bot      The remediation attempt revealed a deeper issue: the
         frontend deployment is entirely missing, so the service
         is down…
```

---

## The Incident: Hallucination on Approve

| Scenario | Observation |
| --- | --- |
| **Expectation** | User clicks Approve -\> Agent executes `patch_deployment` on `gatekeeper-audit`. |
| **Reality** | User clicks Approve -\> Agent detours into `remediation_pipeline` -\> Hallucinates an unrelated "frontend" deployment. |

The patch never ran. This is a story about how `tool_context.session.id` silently broke the approve handshake at the `AgentTool` boundary.

---

## The Smoking Gun in the Audit Log

```json
// Turn 1: Sub-agent request
{"tool": "k8s_health_agent.patch_deployment", "session_id": "60b04298-..."}
{"tool": "orrery_assistant.k8s_health_agent", "session_id": "gchat:spaces/..."}

// Turn 2: User clicks Approve
{"event": "Starting agent run", "session_id": "60b04298-..."}
```

| Context | Session ID Type | Value |
| --- | --- | --- |
| **Chat Thread** | Parent (Persistent) | `gchat:spaces/...` |
| **AgentTool** | Inner (Ephemeral) | `60b04298-...` (UUID) |

The bot re-entered on the **Inner Session ID** — a fresh, empty conversation with no history. Without the context of the Gatekeeper request, the LLM guessed and hallucinated.

---

## The Bug: Session ID Drift

The Google Chat `before_tool_callback` ran on every guarded tool. When it fired, it stashed the `session_id` to allow the click handler to resume execution on Approve.

| Agent Context | `tool_context.session.id` | Handshake Status |
| --- | --- | --- |
| **Root Agent** | Parent GChat Session (Correct) | ✅ Works |
| **AgentTool (Sub-agent)** | Ephemeral Sub-session (Wrong) | ❌ Fails |

For tools inside an `AgentTool` — like `k8s_health_agent.patch_deployment` — the ADK spawns an inner session. The store ended up holding this ephemeral UUID, but the click handler (triggered by the user in the main thread) was looking for the parent session. **The handshake was broken for every specialist agent.**

---

## Why Slack Worked (By Accident)

The Slack bot only wired the callback at the **Root Agent**. Sub-agent tools fell back to text-based `require_confirmation()`. The Google Chat bot, seeking a better UX, wired the callback into every sub-agent, inadvertently exposing this architectural mismatch.

---

## Integrating Google Chat Commands

To make the bot robust, I integrated [Google Chat Commands](https://developers.google.com/workspace/add-ons/chat). Instead of just buttons, it now supports Slash and Quick Commands like `/Approve` and `/Deny`.

### 1. Configuration

I defined these commands in the Google Chat API configuration:

| Command ID | Command Name | Type | Description |
| --- | --- | --- | --- |
| `1` | `/Approve` | Slash/Quick | Approves a pending destructive action. |
| `2` | `/Deny` | Slash/Quick | Cancels a pending destructive action. |

### 2. Handling Interaction Events

When a user types `/Approve`, Google Chat sends a `MESSAGE` event containing the `slashCommand` object. The handler maps the `commandId` to the internal approval logic:

```python
async def on_message(event):
    if "slashCommand" in event:
        command_id = event["slashCommand"]["commandId"]
        if command_id == "1":  # Approve
            return await handle_approval(event)
        elif command_id == "2":  # Deny
            return await handle_denial(event)
```

---

## The Cascade: Why State Propagation Fails

Even with the correct session ID, the handshake would fail because the original callback relied on `tool_context.state` to flag retries. However, `AgentTool` sessions are ephemeral:

1. **Turn 1:** Parent calls Sub-agent. Sub-agent writes `pending=True` to its _inner_ state.
2. **Turn 1 Ends:** Inner session is destroyed. State is lost.
3. **Approve Click:** Parent runner starts.
4. **Turn 2:** Parent calls Sub-agent again. Sub-agent starts with **empty state**, sees no `pending=True`, and blocks again.

**Result:** The bot enters a "Loop of Doom," repeatedly asking for approval because it has no memory of the previous turn's guarded state.

---

## Redesign: The Store Becomes the Source of Truth

The fix moves the entire approve handshake out of ephemeral session state and onto a persistent `ConfirmationStore`. This ensures the "Truth" survives the destruction of sub-agent sessions.

```mermaid
flowchart LR
    subgraph chat["Google Chat interaction"]
        direction TB
        u1["(1) User /Approve"] --> u2["(2) Click handler"]
    end

    store[("STORE<br/>(fingerprint)")]

    subgraph adk["ADK runner execution"]
        direction TB
        r1["(3) before_tool_callback"] --> r2["(4) Check &amp; consume"]
    end

    u2 --> store
    store --> r2
    u2 -->|"re-run agent"| r1
```

_The approve handshake spans two sides: the Google Chat click handler writes a fingerprint to the store, and the before\_tool\_callback on the ADK side checks and consumes it when the agent is re-run_

| Component | Problem | Redesign Fix |
| --- | --- | --- |
| **Session ID Resolution** | `tool_context.session.id` returns the inner UUID. | Reconstruct parent session ID from thread/space stored in state. |
| **Handshake Persistence** | State flags on `tool_context.state` are lost when the sub-agent returns. | The `ConfirmationStore` acts as a persistent, cross-session source of truth. |
| **Approval Logic** | No TTL or argument matching. | Added 120s validity window and `args_hash` matching. |
| **Synthetic Prompt** | Relied on LLM memory that didn't exist in the parent. | Embed original arguments verbatim in the Approve re-entry prompt. |

---

## Implementation Details

**1. Resolving the Parent Session**

```python
def _resolve_parent_session_id(tool_context: Context) -> str:
    state = getattr(tool_context, "state", None) or {}
    thread = state.get("gchat_thread")
    space = state.get("gchat_space", "")
    return f"gchat:{thread or space}" if (thread or space) else "unknown"
```

**2. One-Shot Consumption**

```python
def consume_approved(self, thread_key, tool_name, args_hash):
    # Matches thread, tool, AND arguments within a 120s window
    # One-shot: pop() ensures it can't be reused
    return self._pending.pop(action_id) if valid else None
```

---

## The Secure Re-entry Sequence

When an operator sends `/Approve`, the bot doesn't just "hit enter." It reconstructs the entire context to prevent hallucination through a synthetic prompt.

```mermaid
sequenceDiagram
    actor Operator
    participant Bot as Orrery Bot
    participant Store
    participant LLM

    Operator->>Bot: /Approve
    Bot->>Store: mark_approved(fingerprint)
    Bot->>LLM: synthetic prompt
    LLM-->>Bot: call tool(args)
    Bot->>Store: consume(fingerprint)
    Bot-->>Operator: Success
```

_Sequence of the approve handshake: the operator approves, the bot marks the approval in the store, sends a synthetic prompt to the LLM, the LLM calls the tool, the bot consumes the approval and reports success_

The LLM no longer reconstructs anything. It re-issues the exact call shown on the card.

---

## One UX Polish: A Progress Card On Approve

While I was rewriting the click path, I noticed it had a small papercut: after Approve, the user stared at a silent thread for ~30 seconds while the LLM thought, then a wall of text appeared. The MESSAGE path already posts a live "🔍 Investigating…" card and PATCHes it as the run progresses. The Approve path didn't.

One copy-paste later it does:

```python
progress_message_name = await self._post_initial_progress(
    space_name=pending.space_name, thread_name=pending.thread_name
)
tracker = self._make_tracker(progress_message_name)

result = await self._run_agent(
    session_id=pending.session_id,   # ← now the gchat parent session
    user_id=pending.user_id,
    user_text=synthetic_text,         # ← now embeds args
    user_role=self.resolve_role(pending.user_id),
    space_name=pending.space_name,
    thread_name=pending.thread_name,
    tracker=tracker,
)

await self._update_or_post(
    space_name=pending.space_name,
    thread_name=pending.thread_name,
    message_name=progress_message_name,
    reply={"text": combined_text, "cardsV2": result.get("cardsV2")},
)
```

The user gets the same in-place updating card during Approve as during a normal triage. Same code, same tracker, same debounce.

---

## What "Working" Looks Like Now

End-to-end test, real GKE cluster, real OOMKilled `gatekeeper-audit` pod:

```
You      can you patch deployment in gatekeeper namespace ?
bot      Of course. Which deployment and what patch?
You      list the deployments in gatekeeper-system
bot      [table of gatekeeper-audit (0/1) and gatekeeper-controller-manager (1/1)]
You      can you restart gatekeeper-audit in gatekeeper-system
bot      ⚠️ DESTRUCTIVE: restart_deployment    Send Approve or Deny
You      /Approve
bot      Approved by Taoufiq — executing restart_deployment
bot      Deployment 'gatekeeper-audit' has been restarted.
You      patch memory request and limit on gatekeeper-audit to 1Gi
bot      [describes pod, finds OOMKilled, container=manager]
bot      ⚠️ DESTRUCTIVE: patch_deployment
         namespace=gatekeeper-system, name=gatekeeper-audit,
         patch={spec.template.spec.containers[manager].resources …1Gi}
         Send Approve or Deny
You      /Approve
bot      Approved by Taoufiq — executing patch_deployment
bot      Patch of deployment 'gatekeeper-audit' was successful.
```

Two destructive flows in the same conversation. Both Approves landed on the right tool with the right args. No detours through `remediation_pipeline`. No hallucinated frontend.

---

## Lessons Learned

| Area | Anti-Pattern | Best Practice |
| --- | --- | --- |
| **Session ID** | Trusting `tool_context.session.id` inside an `AgentTool`. | Reconstruct parent session ID from thread/space metadata. |
| **State Handshake** | Using `tool_context.state` for cross-turn approvals. | Use an external `ConfirmationStore` (Postgres/Redis). |
| **Synthetic Prompt** | Asking the LLM to "resume" or "reconstruct" args. | Embed the original arguments verbatim in the prompt. |
| **Visibility** | Relying on print statements or local logs. | Use structured Audit Logs to trace session ID drift. |

---

## A Note on Production Deployments: The Multi-Replica Problem

The code snippets in this article show an in-memory `ConfirmationStore` using a Python dictionary and a threading lock. This is great for a local demo, but it carries a hidden trap for production.

As noted in the [Orrery Deployment Documentation](https://bahalla.github.io/orrery/deployment/), the project is designed for **Kubernetes with multiple replicas**. In a multi-pod environment, an in-memory store is a dealbreaker:

1. **Turn 1:** User clicks `/Approve`. The request hits **Pod A**. Pod A marks the action as approved in its local memory.
2. **Turn 2:** The LLM Runner re-enters the agent. Kubernetes load-balances the request to **Pod B**.
3. **Failure:** Pod B checks its own local memory, finds nothing, and issues a fresh approval card. The user is stuck in the "Loop of Doom" again.

**The Fix:** For production, the `ConfirmationStore` must be backed by a distributed database (like Postgres) or a cache (like Redis). By moving the handshake to the same database that stores your session history, you ensure that any pod in your cluster can validate an approval, regardless of where the original click landed.

---

## What Changed (For Reference)

| Component | Change |
| --- | --- |
| `PendingConfirmation` | Now carries `args`, `args_hash`, `created_at`, `approved`, `approved_at`. |
| `ConfirmationStore` | Adds `mark_latest_approved_for_thread()`, `consume_approved()`, TTL pruning. `pop_latest_for_thread()` still used for Deny. |
| `google_chat_confirmation` callback | Reads parent session id from state. No per-context retry flag. Consults store on every invocation. |
| `handler._handle_app_command_async` | Marks pending approved (no pop), posts a progress card, re-runs in the gchat parent session, updates the card with the final reply. |
| `handler._build_click_synthetic` | Embeds the original args verbatim in the Approve prompt. |
| Approve validity | 120 s after click. Pending TTL: 300 s. Both pruned at lookup. |
| Tests | +5 cases covering parent-session reconstruction, store consumption, stale-approval rejection, unapproved-pending blocking. Total: 618 unit tests, all green. |

---

## Try It Yourself

The bug-and-fix lives in `agents/google-chat-bot/google_chat_bot/confirmation.py`. For more detailed guides and architectural overview, check out the [**Orrery Project Documentation**](https://bahalla.github.io/orrery/).

Two short reads explain everything:

1. `_resolve_parent_session_id` — why state, not `tool_context.session.id`.
2. `ConfirmationStore.consume_approved` — the single primitive that gates retry through three predicates and a time window.

To reproduce locally, run the bot against a cluster that has any `@destructive` Kubernetes tool registered, ask it to patch something, click Approve, and watch the audit log. Before this fix, you'd see two different `session_id` values bracketing the click. After, you'd see one — the gchat one.

---

## What's Next

The same _"don't trust the inner session"_ pattern likely affects any callback we wire across an AgentTool tree — RBAC re-evaluation, metric tagging, audit correlation. The Google Chat bug was the first place it bit, but it won't be the last. Next stop: a small audit of every `before_tool_callback` we register and an explicit test for the AgentTool boundary in each one.

---

_The full source code is on_ [_GitHub_](https://github.com/BAHALLA/orrery)_. Star it if you find it useful._