Building AI Agents for DevOps Part 14: Run Budgets and Fallback Models

In Part 13 the agent learned to read what humans wrote. This post is about the opposite skill: knowing when to stop. Every guard Orrery had was per call. The circuit breaker counts one tool's...

In Part 13 the agent learned to read what humans wrote. This post is about the opposite skill: knowing when to stop.

Every guard Orrery had was per call. The circuit breaker counts one tool's failures. The output cap bounds one result's bytes. RBAC and the confirmation gate decide whether one call may run. None of them can see a loop, and the loops that cost real money are made of calls that each look fine on their own.

Orrery 0.5.0 adds two things: budgets that bound a whole run, and resilience for the one call that had none, the call to the model itself.

A Run Is the Unit That Grows

A run is one agent's work inside one ADK invocation. Each model call in it re-sends the whole run so far, so the cost of a run grows with the square of its length, while the answer at call 90 is rarely better than the one the evidence supported at call 30.

The key is <invocation id>/<agent name>, and both halves matter:

• The invocation separates specialists. An AgentTool specialist gets its own invocation, so its runaway loop is bounded inside the specialist, and the root's turn keeps its own allowance.

• The agent name separates graph nodes. ADK 2.0 runs every Workflow node under the parent's invocation id. Without the name, the five parallel health checkers of the triage sweep would all draw on one budget.

The ledgers are small LRU maps, not session state. They are read on every tool call and must never leak into the persisted state deltas. Evicting an entry can only reset a budget, never let one run's count reach another.

Four Budgets, Four Different Loops

One counter would be simpler. It would also be wrong, because the loops have different shapes:

The delegation loop is the one I did not see coming. A specialist answers "blocked: awaiting approval". The coordinator does not believe it and re-delegates: "you are wrong, the user already approved, execute it now". Every one of those calls succeeds, and every one carries different prose. The breaker counts failures and the repeat guard matches exact arguments, so neither sees anything. The guard counts the one thing they don't: how many times this run called this specialist at all. It is deliberately blunt, because the failure is not subtle. And its refusal tells the coordinator the truth: the specialist already answered, and in this case it was usually right.

The repeat guard refuses the third attempt, not the second. Refusing the second would break the legitimate single retry of a transient fault. The second failure is what tells the two apart: a transient fault rarely recurs verbatim, and one that does is missing something only a human can supply. "The same" means the same tool, the same arguments, and the same error with digits stripped, so a request id or a timestamp doesn't make two identical failures look different. A different error is progress and restarts the count.

It is the only budget scoped to the session, because "try again" spans turns. And like a circuit breaker, it goes half-open after a refusal: someone may have created the namespace in the meantime, so the next identical attempt runs once as a probe.

Refuse So the Run Ends With an Answer

Cutting a run off mid-search leaves the user with nothing. So the call budget works in two steps.

From 70% of the budget, every model request carries a note:

[Tool-call budget: 38 of 50 calls used in this run, 12 left. Stop exploring:
answer with the evidence you already have, say plainly what you could not
check, and suggest how the user could narrow the question.]

It is appended as the last content of the request, never written into a tool result. ADK's after-tool chain stops at the first plugin that returns a value, so rewriting a result would skip every plugin registered after this one. Appending at the end also leaves the cached context prefix untouched.

At the limit, the call is answered without running:

{
  "status": "CALL_BUDGET_EXHAUSTED",
  "error": "This run has made 50 tool calls, its limit ... Answer now with the evidence you already have. Say what you could not check, and suggest a narrower question (one namespace, one service, one time window) rather than retrying."
}

Two rules keep the budgets from fighting the rest of the system:

• Charged after the gates. A call that RBAC or the confirmation gate refused never ran, so it costs nothing.

• Refusals are neutral to the circuit breaker. Every refusal status is on the breaker's ignore list, so a budget refusal can never open a circuit on a healthy tool. A test pins the list to the real status constants, so renaming one fails the build.

The Model Call Had No Resilience at All

Tools had a circuit breaker and retries. The model call had neither. One 503 from the provider failed the turn, and a provider incident was an outage of every replica at once.

These are two mechanisms for two different failures:

• Retry absorbs a blip. Every agent now carries a retry policy for 408, 429 and 5xx, four tries by default. Gemini reads it natively; ADK's LiteLlm maps it to num_retries.

• Fallback survives an outage. Retrying the same model harder does nothing when that model's capacity is what is gone. MODEL_FALLBACK_CHAIN names models to move to.

What does not move the chain matters as much as what does:

• Never on a 400. A malformed request fails the same on every model. Falling back would multiply one user's error by the length of the chain, and bill for each link.

• Never mid-answer. Once a model has started streaming, the turn is committed to it. Splicing a second model onto a first model's partial output gives the user a reply neither model wrote.

• Never across families. ADK records tool calls in history differently for Gemini and for LiteLLM models, so the first failover from one to the other would fail with an unexplained 400. A mixed chain is refused at startup instead.

A 404, on the other hand, does move the chain: that is how a retired model answers, and no retry brings it back. A model that just failed is tried last for 30 seconds, so a turn in the middle of an outage doesn't pay the dead model's whole retry budget on every call.

Silence Is Not Health

Budgets created one new risk. The triage sweep (the batch root that runs on a schedule) always exited 0. If a budget or an RBAC refusal stopped it from reading Kafka, the summarizer reported on the systems it could see, and the CronJob logged a healthy run.

It now exits on what actually happened:

The last line of stdout is a JSON summary with coverage_holes and awaiting_human. A remediation paused at the confirmation gate is listed as awaiting a human. It does not fail the sweep, because waiting for a person is the system working.

Turning It On

The budgets are on by default; each one is disabled with 0. The fallback chain is opt-in:

# run budgets (defaults shown)
ORRERY_MAX_TOOL_CALLS_PER_RUN=50
ORRERY_MAX_DELEGATIONS_PER_RUN=4
ORRERY_REPEAT_GUARD_MAX_FAILURES=2

# model resilience: Pro, falling back to Flash
MODEL_NAME=gemini-3.6-pro
MODEL_FALLBACK_CHAIN=gemini-3.6-flash
ORRERY_LLM_RETRY_ATTEMPTS=3

Keep the retry count modest when a chain is configured: retries run inside each link, so a dead primary costs one full retry budget before the chain moves on. Every failover increments orrery_llm_failover_total{from_model,to_model}.

The design records are AEP-027 and AEP-021.

What's Next

Bounding a run protects the bill and the user's patience. The next post is about protecting something more important: who the agent believes it is acting for, and how to turn security rules that lived in someone's head into checks that fail the build.