Client. A U.S. academic health system, through the application modernization team at a global cloud hyperscaler. The nursing department needs every nurse to start a shift knowing what changed: patient-safety alerts, unit operations news and the merely informational, without reading the whole inbox. The platform is designed for tens of thousands of nursing staff across several campuses.
Approach
I joined a system that already existed and had to be taken to scale. I took over the backend, the MCP server and the multi-agent pipeline when the backend Tech Lead left, and also became the backup for the evaluation framework.
The thesis behind the work is that the cost and risk of an agentic system are designed, not discovered on the invoice. With a language model, every architecture decision is also a pricing decision: what goes into the context, what repeats on every run, how much audio gets generated. If nobody models that before production, the first real data point on cost arrives at month’s end, and with it the uncomfortable conversation.
The problem detected
- Volume multiplies every oversight. Phase one is sized at 78,000 weekly runs and 7,800 concurrent users. A couple of thousand extra tokens per run looks irrelevant in a test; at that scale it is not.
- Much of the prompt repeated verbatim on every run. Guardrails and business rules add up to about 14,500 static tokens, paid at full price every time.
- There is a pricing cliff. Above 200,000 context tokens, the model’s input and output rates double. Phase two adds document ingestion, and left unchecked the context crossed that line.
- Clinical rules were tangled with the arithmetic. The logic of what matters to each nursing role lived alongside the engine that allocates the word budget. Changing a clinical rule meant touching the engine.
Functional analysis
The pipeline classifies each item into tiers before synthesis, and that classification is what later allocates the budget:
| Tier | What it holds | How the briefing treats it |
|---|---|---|
| Tier 1 | Patient safety | Always in, never compressed |
| Tier 2 | Unit operations | In, depending on remaining budget |
| Tier 3 | Informational | Compressed first when space runs out |
| Noise | Emails with nothing actionable | Filtered before reaching the model |
The briefing has a deterministic budget of 500 words, about two minutes of audio. That number is not cosmetic: it is the variable that controls synthesis cost and the time a nurse spends at the start of a shift.
Building the technical solution
- Backend, MCP and agents. FastAPI backend. A Model Context Protocol server integrating Microsoft Graph, SharePoint and BigQuery with On-Behalf-Of delegation: every query runs under the user’s identity, not an all-powerful service account. The Google ADK multi-agent pipeline uses Gemini 2.5 Pro for synthesis and Gemini 2.5 Flash TTS for audio.
- Rules apart from the engine. I separated role-based clinical rules (bedside nurses, clinical specialists, leadership and assistants) from the deterministic word-budget engine, and made the system guardrails immutable contracts. A clinical rule now changes without touching the arithmetic, and the arithmetic is tested without opining on medicine.
- Configuration in one place. I centralized configuration with Pydantic Settings v2. It replaces environment-variable reads scattered across routers, voice helpers and MCP connectors, and validates types at startup, not halfway through a run.
- No silent failures. I removed fallbacks that masked errors and fixed immutable output and citation contracts. In a clinical setting, a briefing that fails visibly beats one that ships incomplete without warning.
- Unblocking infrastructure. The evaluation backend’s Cloud Run deployment was stuck on the container port. I showed the container already took the port as a parameter, so it could be injected from Terraform with no image rebuilds and no duplicated CI/CD pipelines. Technical leadership adopted the fix in the same session. I also diagnosed a frontend 404 behind the load balancer, caused by a build base path that did not match the reverse proxy’s location rules.
Information and data layer
Token economics is, at bottom, governance of what enters the context:
- Context caching. The ~14,500 static tokens of guardrails and rules go through Vertex AI Context Caching, billed at a 90% discount instead of full price on every run.
- Noise filtering. An email filter removes about 8,000 tokens per run that carried nothing actionable.
- Combined result. Together, the two measures cut projected inference spend per batch by 40.8%.
- Clear of the threshold. The same design keeps phase-two context under the 200,000 tokens where pricing doubles.
- Audio audit. In speech synthesis, each generated audio token costs roughly twenty times an input text token, at about 25 tokens per second. The 500-word budget (~700 input tokens, ~3,100 audio tokens) absorbs that gap if the informational tier is compressed first and already-generated audio is cached.
One note on honesty: the 40.8% comes from the financial and load model I co-authored with the Practice Lead, not from metered billing. The platform is not yet running at full scale. The figure states what the architecture costs as designed, which is exactly what needs to be known before scaling it.
How the work was run
- The cost model as an engineering artifact. The load model lives next to the code and is debated as design, not as a spreadsheet finance looks at afterwards.
- Shared evaluation. I co-own automated agent evaluation (an LLM judge over a benchmark of 50 clinical profiles, with telemetry on Cloud Run and BigQuery), so every cost change is checked against briefing quality.
- Operational visibility. I wrote concurrent Python CLIs to audit the engineering backlog and cross-team channels in real time.
- End-to-end documentation. I built a BPMN 2.0 atlas of 12 interactive HTML guides covering the human, clinical and IT flows: delegated authentication, prompt-injection screening, budget allocation and the audio pipeline with its fallback.
What this case proves
- LLM FinOps before production: cost was modeled, attacked at its two main sources (the repeated and the irrelevant) and moved away from the pricing cliff before scaling.
- Resilience by design: rules apart from the engine, configuration validated at startup and contracts that let no error pass silently.
- Technical continuity: I took over a running system midstream and kept evolving it instead of rewriting it.