---
title: "Building AI Agents for DevOps Part 12: Distributed Tracing with OpenTelemetry"
url: "https://tirraflow.com/posts/building-ai-agents-for-devops-part-12-distributed-tracing-with-opentelemetry"
author: "taoufiq"
published: "2026-06-21T12:53:06Z"
updated: "2026-10-15T21:16:04Z"
category: "DevOps & SRE"
difficulty: "intermediate"
series: "Building AI Agents for DevOps"
series_part: 12
series_url: "https://tirraflow.com/series/building-ai-agents-for-devops"
tags: ["Agents", "ADK", "Python", "Observability"]
---

# Building AI Agents for DevOps Part 12: Distributed Tracing with OpenTelemetry

> **Updated for Orrery 0.6.0 (October 2026).** Tracing is unchanged and still enabled by one environment variable. It now has a complement for the person asking rather than the operator: a live trace of specialists and tool calls inside the conversation ([Part 16](https://tirraflow.com/posts/building-ai-agents-for-devops-part-16-streaming-what-the-agent-is-doing)). Traces are for after the fact; the progress trace is for while you wait.

Back in [Part 4](https://tirraflow.com/posts/building-ai-agents-for-devops-part-4-prometheus-metrics) I wired up Prometheus metrics. They answer _how much_ and _how often_ — call rates, error rates, p95 latency, token counts. Useful, but one-dimensional.

Here's where they fall short. A user asks the agent a question, and the turn takes **64 seconds**. The metric tells you it was slow. It does **not** tell you _where_ the 64 seconds went — the routing LLM call? A specialist agent? A tool? The summarization step? For a multi-agent system that fans out across six specialists, "it was slow" is not an answer you can act on.

This post adds **distributed tracing** — and the nice surprise is how little code it takes, because Google ADK already does most of the work.

---

## Three Signals, One Request

Metrics, traces, and logs aren't competitors — they answer different questions about the same request.

![Metrics, traces and logs as three views of the same request](https://tirraflow.com/uploads/44b46b96-e9e5-4268-a31a-84e029d5ad7d.png)

Metrics aggregate. Logs narrate. **Traces are the one that tells you where the time went** — by recording a nested tree of spans (agent → tool → LLM call), each with a start, a duration, and attributes. When all three carry the same `request_id`, you can pivot from a metric spike to the exact trace to the exact log lines.

---

## The Insight: ADK Already Traces

The thing the tutorials bury: **ADK 2.0 is already instrumented with OpenTelemetry.** Every agent invocation, every tool call, every LLM request opens a span under the `gcp.vertex.agent` tracer. In a bare deployment those spans just go nowhere, because there's no exporter configured.

So the job isn't to _create_ spans. It's to:

1. Configure a `TracerProvider` with an exporter, so ADK's spans actually leave the process.
2. **Enrich** those spans with domain context — who the user was, which tool ran, what status came back.

That second point is the one design decision that matters. The naive approach is to make a plugin that opens its own `tool.<name>` span around every call. Do that and you get **duplicate spans** — yours nested inside ADK's, two entries for every tool. The right move is to annotate the span that's _already there_.

---

## The Plugin — Enrich, Don't Duplicate

The whole feature is a `configure_tracing()` bootstrap plus a thin plugin. The plugin never starts a span; it grabs the current one and adds attributes:

```python
class TracingPlugin(BasePlugin):
    async def before_tool_callback(self, *, tool, tool_args, tool_context):
        span = trace.get_current_span()      # ADK's span, already open
        if span.is_recording():
            span.set_attribute("orrery.tool.name", tool.name)
            span.set_attribute("orrery.user_role", tool_context.state.get("user_role"))
            span.set_attribute("orrery.request_id", tool_context.state.get("request_id"))
        return None

    async def after_tool_callback(self, *, tool, tool_args, tool_context, result):
        span = trace.get_current_span()
        if span.is_recording():
            status = result.get("status") if isinstance(result, dict) else None
            span.set_attribute("orrery.tool.status", status)
        return None
```

One trap worth calling out: **do not stash the span object in ADK session state.** Session state is persisted and serialized; a live span isn't serializable, and you'll corrupt the session. Correlation rides on a `ContextVar` and OpenTelemetry's own context — never on state.

Token usage gets the same treatment. ADK already records `gen_ai.usage.*` on the LLM span, so the plugin doesn't re-add it — it just bridges the counts into the existing `orrery_llm_tokens_total` Prometheus metric, so traces and metrics never disagree.

---

## One Env Var, Every Transport

Tracing is opt-in (it needs the `otel` extra) and off by default. The activation is a single environment variable, resolved inside the shared plugin factory:

```python
def default_plugins(*, enable_tracing: bool | None = None, ...):
    if enable_tracing is None:
        enable_tracing = os.getenv("OTEL_TRACING_ENABLED", "").lower() in {"1", "true", "yes", "on"}

    if enable_tracing:
        from .tracing import TracingPlugin, configure_tracing
        if configure_tracing():            # installs the global provider, idempotent
            plugins.insert(0, TracingPlugin())
    ...
```

Because every transport — the Google Chat bot, Slack, the HTTP server, the persistent CLI runner — builds its plugins through `default_plugins()`, flipping `OTEL_TRACING_ENABLED=true` turns tracing on **everywhere at once**. No per-agent wiring. And if the `otel` extra isn't installed, it's a skip-with-warning, not a crash.

---

## Reading the Waterfall

Here's the payoff — a real 64-second turn, exported to Grafana Tempo:

![Grafana Tempo waterfall of a 64-second agent turn, dominated by one 45-second LLM call](https://tirraflow.com/uploads/1ef3b21d-696e-4034-9068-cb6a164a07d6.png)

The span widths make the diagnosis instant. The turn isn't "generally slow" — it's **one 45-second LLM call**, the step that summarizes the Kubernetes pod list. Walk up the tree and the cause is right there: `list_pods` returned ~90 KB of JSON, which ballooned the next model call to 38,000 input tokens.

That's a concrete, actionable fix (paginate or summarize the tool output before feeding it back to the model) that you simply _cannot see_ from a latency metric. This is the entire reason tracing earns its keep.

---

## Logs That Point at Traces

The last piece is correlation. The JSON log formatter now stamps every record with the active `trace_id` / `span_id` and the `request_id`:

```
{"level": "INFO", "logger": "orrery.k8s", "message": "listing pods",
 "request_id": "e-881e8df8", "trace_id": "44fb5a...", "span_id": "9c03b1..."}
```

Grab a `trace_id` off a slow trace, search it in Loki, and you have the exact log lines for that request — and vice versa. The `request_id` works even without the tracing extra installed, so the correlation key is always there.

---

## Try It Locally

The repo ships a one-command stack — Tempo for ingest/storage, Grafana with a provisioned dashboard and datasources:

```bash
make up PROFILES=tracing                             # Tempo :4317, Grafana :3001

OTEL_TRACING_ENABLED=true \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
make run-dev                                         # spans flow to Tempo
```

Open Grafana, pick the Tempo datasource, search `service.name = orrery`, and click any trace to get the waterfall above.

---

## The Takeaway

Two ideas carry the whole feature:

- **Don't reinvent instrumentation your framework already has.** ADK emits the spans; you configure the exporter and add domain attributes. The plugin is ~30 lines, not 300.
- **Enrich, don't duplicate.** Annotate the current span instead of opening a new one. Duplicate spans are worse than no spans — they make the waterfall lie.

Metrics told me a turn took 64 seconds for months. It took one afternoon of tracing to learn that 70% of it was a single oversized LLM call — and to know exactly which tool to fix.

---

_Source code on_ [_GitHub_](https://github.com/BAHALLA/orrery)_. Tracing module:_ [_core/orrery\_core/observability/tracing.py_](https://github.com/BAHALLA/orrery/blob/main/core/orrery_core/observability/tracing.py)_. Plugin wiring:_ [_core/orrery\_core/plugins/_](https://github.com/BAHALLA/orrery/tree/main/core/orrery_core/plugins/)_. ADK observability reference:_ [_adk.dev_](https://adk.dev/)_._