Token Cost Runaway Is an Architecture Problem

Runaway token spend traces back to engineering decisions made during the build.

AI token spend is now the fastest-growing line item in corporate technology budgets. Most teams discover the problem on an invoice, long after the architecture that caused it has shipped to production. Three things drive nearly all of the runaway: context bloat baked into every agent loop, frontier models doing work that cheaper models could handle, and no cost instrumentation until the quarterly review. Each is fixable before the build is done.

• Token costs are not a FinOps problem, they are an engineering discipline, and the fix belongs in the architecture review, not the finance meeting.
• Context re-sent on every agentic loop is the single largest cost driver, not the model tier, not the number of users, not the price per token.
• Teams that apply four architectural levers, prompt caching, model tiering, context compression, and deterministic gates before LLM calls, routinely cut agent costs 50–75% without changing output quality.

The bill that no one planned for

The pattern is now familiar enough to have a name. A team ships an AI feature, a document reviewer, a customer support agent, an internal knowledge tool. The demo is clean. The pilot looks successful. Sixty days later, the infrastructure bill arrives and nobody can explain why it is three times what the model pricing page suggested.

This is not a rare edge case. The average enterprise AI budget grew from $1.2 million per year in 2024 to $7 million in 2026, according to the FinOps Foundation. Enterprise generative AI spending jumped from $11.5 billion in 2024 to $37 billion in 2025. Some Fortune 500 companies are now reporting monthly AI inference bills in the tens of millions of dollars, not for experimental pilots, but for systems that shipped with cost models no one validated against real load.

The cruel paradox is that the per-token price has fallen sharply. Achieving the performance level of an early frontier model dropped more than 280 times in cost between 2022 and 2024. Yet enterprise AI bills have risen by an estimated 320% over the same window. Cheaper per token does not mean cheaper overall when the number of tokens per task multiplies faster than the price drops.

When a team builds an agentic workflow without modeling token consumption per loop, the architecture locks in the cost, and no budget meeting can fix it after the fact.

Three root causes behind every runaway bill

1. Context accumulation on every loop. Every LLM API call is stateless. Agentic frameworks compensate by re-sending the full conversation history on every turn. By step 20 of a typical agent loop, the input alone can exceed 50,000 tokens, and re-sent context accounts for roughly 62% of the total bill. One team of 20 developers using an agentic coding tool, without caching, can burn $110,000 per month on a single toolchain.
2. Frontier models doing commodity work. Most agentic frameworks default to a single model for every step, orchestration, tool selection, data retrieval, and final synthesis alike. Organizations that route every workload to frontier models pay up to $18.40 per million tokens. Those running tiered architectures pay a blended $2.31 per million tokens for similar results. That 8x spread lives entirely in routing logic, not in model capability.
3. No cost architecture at build time. Token cost modeling is treated as a post-launch FinOps task, not an engineering requirement. The teams that avoided budget surprises shared one characteristic: they modeled token volume per workflow type before the architecture was finalized, not with precision, but with a realistic loop count and context depth. The teams that skipped that step are the ones reconciling unexpected spend after the fact.

These three causes share a common mechanism: they all defer cost decisions to the runtime. Context accumulation grows invisibly until production load exposes it. Model uniformity is the path of least resistance in every SDK. Cost instrumentation requires intentional engineering that nobody put on the sprint. Together, they guarantee that the economics of any non-trivial agentic system will surprise the team that built it, unless the architecture was designed with token spend as a first-class constraint from day one.

What we've learned from 200+ engagements

For more than two decades we have built and scaled production software for clients across healthcare, finance, education, retail, and technology. Long before agentic AI, the disciplines that kept those systems affordable were the same ones that matter now: model the cost of an expensive call before you make it, push work to the cheapest layer that can do it correctly, cache what repeats, and instrument spend so it never surprises you. Token economics did not invent these principles, it reiterated them. The teams who treat AI spend as a new technology problem never solve it; the teams who recognize it as the same engineering fundamentals applied to a new kind of expensive call do. In practice that means token cost modeling belongs in the same conversation as database schema and API rate limits: early, explicit, and owned by the engineers doing the build, not the finance team reading the bill.

Our Build / Boost / Buy framework applies directly to this problem. When a workflow is differentiating, when the specific sequence of steps, the data, and the decision logic is what makes the product valuable, you build custom and you instrument every LLM call from the first prototype. When you are boosting an existing tool, you focus on last-mile fit: right-sizing the model, trimming the context, and adding a deterministic gate so the LLM only sees the work it actually needs to reason about. When you buy off-the-shelf, you govern consumption the same way you govern cloud infrastructure, with budgets, alerts, and chargeback by team. The 30-Day AI Enablement Workshop is where we do this work in practice: forward-deployed with the client team, mapping the actual workflow, measuring token consumption on real tasks, and shipping a working artifact with cost visibility built in before we leave.

The cheapest LLM call is the one you don't make, and the most expensive architecture decision is the one you defer to the invoice.

What to do instead: five architectural moves that move the bill

1. Add deterministic gates before every LLM call. Before routing a task to a model, ask: does this step actually require inference? Deterministic logic, rule checks, regex filters, structured lookups, relevance classifiers, handles a large share of the decisions an agent loop takes, at a fraction of the cost. GitHub's engineering team found that moving deterministic data-gathering reads out of the agent reasoning loop produced a 62% token reduction on one workflow and a 43% reduction on another. Nothing was removed from the output. The LLM was simply no longer doing work that code could do. Success signal: LLM call rate drops without output degradation.
2. Implement prompt caching on every repeated system prompt. Cached tokens are billed at 10–25% of normal input cost across all major providers. Any system prompt, tool definition block, or static document that is re-sent across multiple agent turns should be marked for caching. For an enterprise running 5,000 agent loops per day, prompt caching alone saves over $2,000 per day. Semantic caching, where a cache layer intercepts semantically similar queries and returns stored responses without a fresh LLM call, has demonstrated up to 73% cost reduction in high-repetition workloads. This is the single highest-leverage change most teams can make without touching the application logic. Success signal: Cache hit rate above 60% for recurring workflows.
3. Tier your model routing by task complexity. A frontier model is the right tool for synthesis, complex reasoning, and final-answer generation. It is the wrong tool for tool selection, classification, retrieval scoring, and intermediate data formatting. A workflow that runs 80% of its steps on a capable lightweight model and escalates only the hard 20% to a frontier model costs roughly 12% of an all-frontier workflow, with similar end results. For multi-agent systems, use a hierarchical architecture: budget models for worker agents, frontier models only for the lead orchestrator, a structure shown to achieve 97.7% of full-frontier accuracy at approximately 61% of the cost. Success signal: Blended cost per million tokens below $3.
4. Compress context, trim tools, and summarize history. Context is not free. Tool definitions alone can consume 40–50% of a context window; Anthropic's own tooling has observed 55,000 to 134,000 tokens of tool definitions in a single unoptimized prompt. Trim the tool list to what is relevant for the current step. Compress conversation history with a running structured summary rather than re-sending raw turns. Compressing context aggressively, from 10,500 tokens per request to 650 in one documented case, represents a 94% token reduction per call. The model does not need the most information; it needs the right information. Success signal: Average tokens per agent call cut by half from baseline.
5. Instrument token spend at the workflow level from day one. Tag every API call with workflow type, step name, and user or team identifier. Track tokens-per-call alongside call count, raw token totals confuse workload variation with efficiency change. Set per-workflow cost budgets and wire anomaly alerts before launch, not after. The teams that avoided budget surprises modeled token volume per workflow type during architecture design, not post-launch. By the time a production system has locked its architecture, the cost is baked into the user experience and optimizing it requires rebuilding. Success signal: Cost-per-workflow visible in real time, not quarterly.

What this looks like in practice

Illustrative example: autonomous document-review agent

The following is a hypothetical, illustrative example. Picture a growth-stage SaaS company running a custom autonomous document-review agent, one that reads contract attachments, flags compliance gaps, and drafts revision summaries. It works well in demos and early production. Then, by month four, the monthly inference bill has tripled from the initial estimate. Usage didn't grow; the agent simply re-sends the full contract text plus accumulated conversation history on every review turn. A single complex contract ends up generating 40,000-plus input tokens per step by the fifth iteration. This is the shape the runaway almost always takes.

Now apply the five moves. A pre-processing gate extracts only the flagged clauses into a structured payload, cutting context per call by roughly 78%. The document-fetch and metadata-lookup steps move to deterministic code before the LLM is invoked at all. Prompt caching is enabled on the 3,000-token compliance ruleset that was being re-sent on every call. The workflow is tiered: a lightweight model handles clause classification, and the frontier model handles only final summary drafting. In a scenario like this, monthly inference cost can fall on the order of two-thirds, and output quality typically holds or improves, because the model is no longer distracted by irrelevant context.

How to start without making an expensive mistake

Five things the team responsible for an AI build can do this week:

1. Audit one workflow end-to-end. Pick the highest-volume agentic task in production (or in staging). Log every API call with token counts by step. You will almost certainly find that 2–3 steps account for more than 70% of the bill, and at least one of them does not require LLM inference at all.
2. Enable prompt caching today. If your system prompt exceeds 1,000 tokens and it repeats across calls, caching it is a one-line change in most SDKs that immediately cuts the cost of that portion by 75–90%.
3. Map every step to a model tier. For each step in your agent loop, answer: does this require frontier reasoning, or would a lighter model at 10% of the price produce an acceptable result? Route accordingly. The first time you do this mapping, you will find steps where the question answers itself.
4. Set a cost budget per workflow, not per month. A monthly aggregate cap does not tell you which workflow is burning money. A per-workflow budget with an alert threshold tells you within hours. Wire this before launch, even if the budget is generous, you are establishing the instrumentation, not the constraint.
5. Write a should-cost model before the next build starts. Estimate: number of loops per task, tokens per loop at each step, model tier per step, calls per day at projected load. Multiply. If the number surprises you at planning time, it will devastate you at invoice time. Treat the should-cost model as a first-class architecture artifact, not an afterthought.

Sources

1. "AI tokens: How to navigate AI's new spend dynamics", Deloitte Insights, deloitte.com, accessed 2026-06-19.
2. "AI Inference Cost Crisis 2026: Why Your AI Bill Is Exploding", Oplexa / FinOps Foundation 2026 State of FinOps Report, oplexa.com, accessed 2026-06-19.
3. "AI Agents Burn 50x More Tokens Than Chats", LeanOps (analysis of 30 enterprise agent audits, Mar–May 2026), leanopstech.com, accessed 2026-06-19.
4. "Improving token efficiency in GitHub Agentic Workflows", GitHub Engineering Blog, github.blog, accessed 2026-06-19.
5. "How I Reduced LLM Token Costs by 90% Building AI Agents", Towards Data Science / Medium, medium.com, accessed 2026-06-19.
6. "LLM Token Optimization: Cut Costs & Latency in 2026", Redis Engineering Blog, redis.io, accessed 2026-06-19.
7. "AI Token Cost Enterprise: Stop Budget Blowouts in 2026", Elvex (13x token consumption growth since Jan 2025 dataset), elvex.com, accessed 2026-06-19.