Agents in the Open Part 5: Stopping Without Cutting an Agent Off

The first four parts were about what the Orrery Lab must never do. This one is about what it must always finish. An agent's turn isn't a quick request: it can call five tools over thirty seconds, and...

The first four parts were about what the Orrery Lab must never do. This one is about what it must always finish. An agent's turn isn't a quick request: it can call five tools over thirty seconds, and one of them may be halfway through when you deploy. Orrery 0.6.6 fixed three reliability gaps that only show up once something is running for other people. Each one was a default that looked fine.

A Turn Outlives Its Client, So Shutdown Must Wait For It

Since Part 16, a streamed turn keeps running when the browser tab closes. That's deliberate: a closed tab must not cut a tool call in half. Orrery keeps a strong reference to each running turn, because asyncio holds only a weak one, and a task nobody awaits can otherwise be garbage-collected mid-run.

The review found the other half was missing. On SIGTERM, uvicorn waits for connected requests. A turn whose client had left had no connection, so nothing waited for it, and a rolling deploy ended it mid-tool: exactly what keeping it alive was meant to prevent.

Shutdown now has two phases:

finally:                                   # FastAPI lifespan, on SIGTERM
    await drain_turns(cfg.shutdown_grace_seconds)   # default 30 s

• Stop taking new work. On SIGTERM, uvicorn closes its listening socket, so new connections are refused and a load balancer moves on. Orrery also has its own guard that answers new turns with 503 and Retry-After, but the review showed it is raised too late to matter: it is set after uvicorn has already stopped accepting. The fix still to ship is to raise it from the signal itself and fail /readyz at the same moment.

• Finish what started. Wait up to the grace period for turns still running, then log how many were cut off. A turn that ends early is reported, not hidden.

The Timeouts Must Fit Inside Each Other

A grace period only helps if the layer above allows it. There are three nested clocks, and each must fit inside the one around it:

• uvicorn: --timeout-graceful-shutdown 15 for connected requests;

• Orrery: ORRERY_SHUTDOWN_GRACE_SECONDS=30 for turns whose clients left;

• the platform: Docker's stop_grace_period: 50s in the lab, or Kubernetes' 60 seconds minus a 10-second preStop in the Helm chart.

Docker's default is 10 seconds. With it, all of the above would end in SIGKILL long before the drain finished, and nothing would log why. When you add a grace period anywhere, check the one around it.

The Default Command Is a Production Decision

While adding the drain, I followed where it would actually run, and found something worse. The published image's default command, and the Helm chart's, was adk web. That is Google ADK's developer UI. It ignores AUTH_ENABLED and the JWT_* settings, and lets anyone read any user's sessions. A deployment that relied on the defaults had none of Orrery's front door: no sign-in, no rate limits, no verified approvals, no CSP, no drain.

The lab was never exposed, because its Compose file named its own command. That's luck, not design. So in 0.6.6:

• the image and the chart run uvicorn orrery_assistant.app:api, which refuses to start without JWT settings;

• the chart's auth.enabled: false now means an anonymous, viewer-only front door instead of a developer UI;

• the chart's probes moved to the HTTP port, where /healthz and /readyz really are (the old port served nothing);

• the Compose demo profile binds to loopback, and adk web logs an error when started with AUTH_ENABLED=true.

A default is what runs when nobody made a decision. It should be the configuration you would choose for a stranger's production.

"Incomplete" Is Not "Healthy"

The third gap was in how a scheduled health sweep reports its result. A sweep runs one checker per system and then a summarizer that records a verdict. Before these fixes, a checker that reported nothing, or a run with no system to check at all, could end as complete and healthy. Silence was read as good news.

Now the run records, for each system, whether its checker reported at all, and the exit code says what happened:

• 0: every system was checked, and a verdict was recorded;

• 1: failed. The run raised, or produced no verdict at all;

• 2: incomplete. A checker was silent, a read was refused, the verdict had to be inferred from text, or there was nothing to check.

A cron job or a CI step can tell "all good" from "didn't look". And with no specialist able to remediate (the lab has none), the workflow has no remediation loop at all. The report says a human must act, instead of trying anyway.

The Pattern

• Work that outlives its request needs its own shutdown phase: stop accepting, then drain, then report.

• Nested timeouts must fit inside each other. Check the platform's, not just your own.

• Ship defaults you would run in production. Someone will.

• Report "I didn't check" as a separate outcome from "I checked and it's fine".

The lab now runs Orrery 0.6.6, verified and pinned by digest. The lab isn't open to readers yet (only Tirraflow admins can sign in), and I will update Part 1 when that changes. Until then, the same stack runs locally: make install && make demo from the Orrery repository, with your own model API key, seeds the same Kafka incident and opens the console on localhost:8000.