Building AI Agents for DevOps Part 8: Agent Evaluations

In Part 7, I refactored cross-cutting concerns into ADK Plugins and converted all tools to async. The platform is clean, fast, and safe. But one question remained unanswered: does the agent actually...

In Part 7, I refactored cross-cutting concerns into ADK Plugins and converted all tools to async. The platform is clean, fast, and safe. But one question remained unanswered: does the agent actually call the right tool?

425 unit tests verify that each tool returns correct data when called directly. But no test verifies that the LLM, given "Is my Kafka cluster healthy?", chooses get_kafka_cluster_health instead of list_kafka_topics or describe_consumer_groups. Change an agent's instruction prompt and you could silently break tool routing without a single test failing.

This post adds agent-level evaluations — 22 scenarios across 4 agents that verify tool selection with a real LLM.

Unit Tests vs Agent Evals

The distinction is simple:

Unit test:  call get_kafka_cluster_health() directly → assert result is correct
Agent eval: send "Is my Kafka cluster healthy?" to the agent → assert it chose get_kafka_cluster_health

Unit tests catch broken tool implementations. Agent evals catch broken tool routing — the LLM choosing the wrong tool, passing wrong arguments, or calling extra tools unnecessarily.

How ADK Evaluation Works

ADK provides AgentEvaluator, which takes three inputs:

1. The agent module — loaded and run with a real LLM

2. A test dataset (.test.json) — user queries with expected tool trajectories

3. Criteria (test_config.json) — score thresholds that must pass

The evaluator sends each query to the agent, records which tools the LLM calls, and compares the actual trajectory against expected. Here's the flow:

User query → LLM → tool call(s) → compare against expected trajectory
                                        ↓
                              tool_trajectory_avg_score = 1.0 or 0.0

Anatomy of an Eval Scenario

Each scenario is a JSON object with a user message, expected tool calls, and a reference response:

{
  "eval_id": "check_cluster_health",
  "conversation": [
    {
      "invocation_id": "inv-001",
      "user_content": {
        "parts": [{"text": "Is my Kafka cluster healthy?"}],
        "role": "user"
      },
      "final_response": {
        "parts": [{"text": "The cluster is healthy with 2 brokers online."}],
        "role": "model"
      },
      "intermediate_data": {
        "tool_uses": [
          {"name": "get_kafka_cluster_health", "args": {}}
        ],
        "intermediate_responses": []
      }
    }
  ],
  "session_input": {
    "app_name": "kafka_health_agent",
    "user_id": "test_user",
    "state": {"user_role": "viewer"}
  }
}

The key field is tool_uses — this is the expected trajectory. The evaluator checks that the LLM called exactly these tools with exactly these arguments.

The Test Runner

Each agent gets a pytest file that mocks external dependencies and runs the evaluator. The pattern is the same across all agents:

@pytest.mark.eval
@pytest.mark.asyncio
async def test_agent_eval():
    if not _has_llm_credentials():
        pytest.skip("Requires Gemini credentials")

    with patch("kafka_health_agent.tools._get_admin_client",
               return_value=_make_admin_client()):
        await AgentEvaluator.evaluate(
            agent_module="kafka_health_agent.agent",
            eval_dataset_file_path_or_dir=EVAL_DIR,
            num_runs=1,
        )

The mock is critical: the LLM calls the real tool function, which calls the mocked client, which returns fixture data. The agent thinks it's talking to a real Kafka cluster.

Why Trajectory-Only Scoring

ADK offers several evaluation criteria. I only use one:

{
  "criteria": {
    "tool_trajectory_avg_score": 1.0
  }
}

I intentionally excluded response_match_score (ROUGE-1 text similarity). Here's why: the same correct answer phrased differently scores 0.0 on ROUGE. Run the eval twice and you get different scores because the LLM words its response differently each time.

Expected: "The cluster is healthy with 2 brokers online."
Actual:   "Your Kafka cluster has 2 active brokers. Everything looks good."
ROUGE-1:  0.25 — FAIL (despite being perfectly correct)

Tool trajectory is deterministic and tests what actually matters: did the agent call the right tool with the right arguments?

Coverage

22 scenarios across 4 agents:

$ make eval
agents/docker-agent/tests/test_docker_eval.py::test_agent_eval PASSED
agents/kafka-health/tests/test_kafka_eval.py::test_agent_eval PASSED
agents/k8s-health/tests/test_k8s_eval.py::test_agent_eval PASSED
agents/observability/tests/test_observability_eval.py::test_agent_eval PASSED
4 passed in 68.12s

Lessons Learned

Writing agent evals is different from writing unit tests. The LLM is a non-deterministic component, and several assumptions I had were wrong.

1. Prompts must include all context

"Describe the web-app-1 pod" seems clear to a human. But the LLM asks "which namespace?" instead of calling the tool. Fix: "Describe the pod web-app-1 in the default namespace."

2. Expected trajectory must match agent instructions

The K8s agent's instruction says "Start with get_cluster_info and get_nodes for an overview." When a user asks for a cluster overview, the LLM follows this instruction and calls both tools. If the expected trajectory only lists get_cluster_info, the eval fails — even though the agent did the right thing.

3. Include all arguments the LLM will pass

If the user says "in the default namespace", the LLM passes namespace: "default" explicitly, even though it's the tool's default value. The expected args must include it.

4. Avoid ambiguous tool names

"Are there firing alerts in Prometheus?" caused the LLM to call get_active_alerts (Alertmanager) instead of get_prometheus_alerts (Prometheus). Both are reasonable interpretations. The prompt must be unambiguous.

5. Mock data format must match the tool

Docker tools use --format json. My initial mock returned human-readable table output. The tool called json.loads() on it and crashed. Fixture data must match the real format.

CI Integration

Evals are gated behind @pytest.mark.eval so they don't run on every push (they need LLM credentials and take ~60 seconds). The CI workflow has a dedicated eval job triggered manually or via the run-eval PR label:

eval:
  runs-on: ubuntu-latest
  if: github.event_name == 'workflow_dispatch' ||
      contains(github.event.pull_request.labels.*.name, 'run-eval')
  steps:
    - uses: actions/checkout@v6
    - run: make eval

Regular make test runs 425 unit tests in ~1 second and skips evals entirely. No LLM costs on every commit.

What's Next

The current evals cover individual agent tool routing. Two gaps remain:

• Orchestrator routing — does the devops-assistant delegate "check Kafka lag" to the Kafka agent (not K8s)?

• Hallucination detection — does the agent report a cluster as healthy when the mock data says it's down? ADK's hallucinations_v1 criterion could catch this, but it requires the Vertex Gen AI Evaluation Service.

Source code on GitHub. Eval files: agents/kafka-health/tests/evals/. AEP-002: docs/enhancements/aep-002-agent-evaluation.md.