Datagras
Back to blog

FinOps12 July 2026

Taming the Token Explosion: Controlling the Cost of Agentic AI

A team ships an AI agent that can look things up, call a few internal tools, and double-check its own work before responding. It works well in the demo. Two weeks after it ships, someone in finance asks why the AI line item just tripled, and nobody on the engineering team can point to a single cause. It wasn't one thing. It was twenty small things, all happening inside a loop nobody was watching closely.

That's the defining cost problem of agentic AI in 2026, and it's different in kind, not just degree, from anything FinOps teams dealt with in the chatbot era.

Why agents cost so much more than a simple query

A standard chatbot interaction is one call: prompt in, response out. An agent doesn't work that way. It plans, calls a tool, reads the result, decides whether it needs more information, calls another tool, reasons over the combined output, and only then responds. Every one of those steps is a separate call to the model, and because the model has no memory between calls, each one resends the entire conversation history up to that point. By the twentieth step in a long-running loop, a team can be paying for the same system prompt and conversation history twenty separate times.

EY's 2026 analysis of agentic AI costs put a number on this shift: a simple customer-service interaction cost about $0.04 in 2023. A comparable interaction routed through a modern orchestrated agent, with tool calls and reasoning loops, costs around $1.20 today, roughly 30 times more, even though the price of a single token has fallen sharply over the same period. The bill goes up because the number of tokens per task goes up faster than the price per token goes down.

Where the waste actually lives

Most teams assume the fix is a cheaper model. That's rarely the biggest lever. The bigger cost driver is architectural: how many times the agent calls the model, how much context gets resent on every call, and how much of that context is actually necessary.

A few patterns show up repeatedly in cost audits of production agent workloads:

  • Re-sent context. Research on agentic cost attribution has found that resent conversation history can account for well over half of total inference spend on long-running agent tasks.
  • Over-provisioned tool lists. Agents given access to a large catalog of tools burn tokens evaluating which ones to use, even when most tasks only ever need two or three.
  • Frontier models on easy tasks. Sending routine, low-complexity requests to the most expensive available model is one of the most common and easiest to fix patterns in production systems.

The levers that actually work

LeverWhat it doesTypical savings
Model routingSends easy tasks to a cheap model, escalates only when needed30-80%
Prompt cachingAvoids re-paying for repeated system prompts and context40-90% on input tokens
Context trimmingCuts unnecessary history and tool definitions from each call30-60%
Hard budgets and alertsCatches runaway loops before they become a monthly surprisePrevents outlier spend, not baseline savings

Combined, model routing, prompt caching, and context optimization together commonly bring total agentic cost down by 60% to 80% in production deployments, without a noticeable drop in output quality. None of these require switching model providers, which usually matters less than teams expect. On a multi-step agent workflow, swapping one frontier model for another rarely moves the total bill much. Cutting an unnecessary step out of the loop does.

Budgets are moving from the team level to the individual level

Token governance is starting to show up in a place FinOps hasn't traditionally operated: the individual employee. At Nvidia's GTC 2026 conference, CEO Jensen Huang reportedly described issuing engineers token budgets worth roughly half their base salary, treating heavy usage as a sign of engagement rather than a cost problem to police. Other companies have taken the opposite approach, capping individual token consumption the way they'd cap a corporate card.

Whichever direction an organization leans, the underlying shift is the same: the token is becoming a resource that gets budgeted, monitored, and occasionally throttled, the way compute and storage already are. Building that muscle now, before agentic workloads triple in volume, is considerably cheaper than building it after the invoice forces the question.

Instrument before you optimize

None of the levers above work without visibility into where the tokens are actually going, call by call, agent by agent. Teams that jump straight to optimization without that visibility tend to guess wrong about where the waste is.

That visibility, down to the session, the agent, and the task, is what SAVI gives engineering and finance teams before the monthly bill becomes the first place anyone notices a problem.