Client. The application modernization team at a global cloud hyperscaler, which builds AI agents to migrate, diagnose and optimize its customers’ systems, and needs a verifiable criterion for deciding which of those agents reach production.
Approach
This is the first case in the portfolio where the fourth link —AI over the data— is not the instrument I work with but the delivered product. It spans three links: software engineering, data engineering and AI. The distinction from earlier cases is deliberate: what gets built here is the infrastructure another team uses to evaluate its own agents.
The thesis that governs the work is that an agent is not promoted to production on an impression. A convincing demo is not evidence; an isolated LLM metric is not either, because it says nothing about business impact or about whether the automated judge agrees with a domain specialist. For the decision to be defensible, three pieces have to be chained: evidence captured under a stable contract, a judge calibrated against a human expert, and a deterministic rule that turns that evidence into a promotion decision. Without all three, agent governance stays in the territory of opinion.
The problem detected
Evaluating an agent is different from evaluating a model. The agent calls tools, mutates memory and chains turns; the final answer is only the visible tip of a path that may have broken at any point. Two practical consequences:
- Evidence stays tied to the runtime. Each agent framework records its steps its own way, so runs stop being comparable across frameworks and across versions. With no comparability there is no measurement, only anecdotes in chronological order.
- The technical metric never reaches the decision table. A quality score does not answer the question the decision-maker asks: if we ship this version, what changes per 1,000 interactions, and with what uncertainty?
The stated problem was measuring agents. The real problem was prior to that: there was no evidence contract that survived a change of runtime, and no formal bridge between the judge’s metric and the release decision.
Functional analysis
Before writing the evaluation engine I defined what evidence is and at how many levels it is captured. The contracts are immutable and typed with Pydantic v2, so evaluation is decoupled from the agent runtime and traceability stays comparable across runs:
| Evidence level | What it captures | What it enables |
|---|---|---|
OUTPUT | The agent’s final answer | End-to-end quality comparison across versions and across agents |
STEPS | Tool calls and memory transitions along the path | Root-cause diagnosis: which segment broke, not just that it broke |
TELEMETRY | Multi-turn OpenTelemetry traces of the execution | Temporal correlation, performance and reconstruction of the full run |
Fixing the three levels first, and only then the engine that consumes them, is what keeps the platform from being coupled to whichever agent framework was fashionable the month it was written.
Building the technical solution
The platform runs, observes, compares and evaluates agents. The pieces:
- Execution and observability. I integrated the Google ADK runner with OpenTelemetry instrumentation, capturing tool calls, memory transitions and multi-turn traces. Quality evaluation runs on Vertex AI Gen AI Evaluation Service.
- Value engine. I built the component that converts technical LLM metrics into estimated impact per 1,000 interactions and per segment. The estimate is not published as a bare number: it carries a stratified Monte Carlo bootstrap of 1,000 samples and a 90% confidence interval, because an estimate with no uncertainty band is an assertion dressed up as data.
- Promotion gate. On top of that engine runs a deterministic four-state tree —
BLOCKED,UNCALIBRATED,HOLD,PASS— traversed in order, which governs agent promotion. Deterministic is the point: the same evidence always produces the same decision, and the decision can be audited afterwards. - Backend and performance. Async REST gateway in FastAPI with async SQLAlchemy to orchestrate concurrent benchmarks. I removed I/O bottlenecks with in-memory caching and O(1) lookups in dataset comparison and reading.
- Inspection interface. React 18 / Vite dashboard with real-time telemetry, memory-delta inspection, tool-invocation correspondence and comparative radar charts across agents. The critical inspection and run-management flows are covered by 27 frontend unit tests.
- Deployment. Containerized with a multi-stage build and deployed to Cloud Run via Cloud Build, with IAM-authenticated access.
Information and data layer
Data governance is the substance of the product here, not an appendix:
- Calibrating the judge. I implemented an inter-annotator agreement service with chance-corrected Cohen’s kappa and confidence bands, to audit how far the LLM-as-a-judge evaluation agrees with a domain specialist’s baseline. An automated judge nobody contrasted against a human is a number generator, not an evaluator.
- Residency and minimization. Execution is restricted to a defined region and the flow includes PII sanitization before data reaches the evaluator.
- Continuity without the network. Deterministic fallbacks keep testing and operational continuity running with no dependency on external services, so the suite does not hinge on an endpoint’s availability.
- Reference sets. I prepared golden datasets for three families of modernization cases: Java/Spring Boot migration to Cloud Run, Cloud SRE diagnosis and SQL/BigQuery optimization. With no stable reference set there is no comparison across versions, only the feeling that this one runs better.
One note on honesty: the impact figures the engine produces are modeled estimates derived from technical metrics, not savings audited by a third party. The platform exposes the confidence interval precisely so that nobody reads them as anything else.
How the work was run
- Quality as a gate, not an intention. The pipeline runs a five-stage quality gate with more than 150 backend and frontend tests, static analysis and strict TypeScript compilation. What does not clear the five stages does not reach the deployed environment.
- Specification before code. I wrote the practice’s playbook in English for setting up agentic projects with Spec-Driven Development: the same criterion I apply across the other cases —fix the contract before the implementation— turned into a document other teams use.
- Review and mentoring. Technical reviews and support for other engineers on concurrency, data contracts and specification writing.
What this case proves
- AI as a deliverable, not an instrument: the method’s fourth link built and shipped, with contracts, calibration and a gate.
- Governance made operational: the decision to promote an agent lives in a deterministic, auditable tree, not in a committee watching a demo. It is the same governance-as-infrastructure thesis I hold across the portfolio, applied to the hardest object to govern.
- Full platform engineering: typed contracts, async backend, observability, an inspection frontend, containerization and cloud deployment, with a quality gate that makes all of the above verifiable.