For two years, cost per token was a design constraint we could feel. It decided how many candidates a re-ranker saw, how much of a document survived into the context, whether a verification pass was affordable, how many steps an agent could take before someone asked what this was costing.
That constraint has largely dissolved. Frontier-class inference has fallen by roughly an order of magnitude on the timescale that matters to an architecture, and the cheaper tiers have fallen further. Most of the decisions we made to conserve tokens are now decisions about nothing.
The interesting part is not the savings. It is what became the limit instead.
What we did first, and why it was wrong
The reflex was to spend the surplus: more retrieved candidates, more verification passes, more agent steps, longer context. Every one of those individually improves a benchmark. Taken together they made two things worse, and neither showed up on a cost dashboard.
Latency stopped being incidental. Cost per call fell; time per call did not fall proportionally, and the number of calls went up. A pipeline that got a second verification pass and a wider candidate set is not marginally slower — it is slower in a way users describe as broken. Nobody approved that trade, because nobody was asked: the change was justified on quality and priced on tokens, and the latency budget was not in the argument.
More calls means more failure surface. Each additional tool call and model call is another place to time out, another chance to return something malformed, another opportunity to continue with a corrupted context. Reliability is multiplicative down a chain. Adding steps because they are cheap adds failure modes at the same rate.
Where the constraint actually moved
Three constraints replaced the one that dissolved, and they are harder to see because none of them appears on an invoice.
Latency budget. It is now the scarce resource in most interactive paths, and it should be allocated the way cost used to be: written down, divided among stages, and defended. A stage that wants a second pass has to take the time from somewhere.
Context discipline. Cheap tokens removed the financial penalty for over-including, but not the quality penalty. When including everything was expensive, the bill enforced curation. Now the only thing enforcing it is a decision to keep curating.
Attention, in the human sense. More steps in a pipeline is more surface for an operator to understand during an incident. We have stopped treating pipeline complexity as free just because compute is.
What we re-architected
- The latency budget is a written artefact, allocated per stage, reviewed when a stage wants more. Same discipline the token budget used to get.
- Extra passes must earn their place against the trajectory, not the benchmark. A verification pass that improves a score but doubles the failure surface is a bad trade at any price.
- Routing questions changed shape. They used to be mostly “what is the cheapest model that clears the bar.” Now they are about latency, context handling, and behavioural stability under repeated runs. Price is a tiebreaker, not the criterion.
- Batch and interactive paths diverged. When tokens were expensive, both were built to the same frugal design. Now offline paths can afford breadth that interactive paths cannot, and pretending they are the same system costs quality in one direction and latency in the other.
The general lesson
When a constraint disappears, the systems designed around it do not become correct. They become unconstrained, which is not the same thing, and their new limits are the ones nobody has instrumented.
The useful move after a price collapse is not to spend the difference. It is to go back to the traces and find out which quantity is scarce now — and then be as disciplined about that one as the old bill made us about tokens.