The invoice is the wrong feedback loop
An AI feature that works in a demo and works at 10,000 requests a day are two different systems. The difference is rarely quality — it is economics. Teams that ship without cost instrumentation learn their unit economics thirty days late, aggregated into a single number that cannot be attributed to a feature, a customer, or a prompt.
We treat token cost the way infrastructure teams treat latency: measured at the call site, attributed to an owner, and visible on a dashboard the same day.
What we log on every inference call
Every model call passes through a single logging boundary rather than being scattered across feature code. At that boundary we capture:
- Model and provider — including the resolved version, not the alias
- Input and output tokens — separately, because their prices differ by an order of magnitude
- Estimated cost — computed at log time from a versioned price table
- Latency — end-to-end and time-to-first-token
- Attribution — user, session, agent, and tenant
- Status and failover chain — which model actually answered, and what it fell back from
The failover chain matters more than it appears. A silent downgrade from a frontier model to a cheaper one changes both your cost curve and your quality curve, and without the chain recorded you cannot explain either.
Cost is a routing input, not a report
Once cost is available at runtime, it stops being a report and becomes a control signal. The routing layer can consider it directly:
- Classification, extraction, and formatting go to small models by default
- Long-context synthesis and multi-step reasoning are the only work that earns a frontier model
- A tenant approaching a quota triggers an upgrade prompt rather than an unbounded bill
The important architectural property is that this logic lives outside the model. The model generates. The routing layer decides whether it should have been called at all.
Three habits that reduce spend without reducing quality
Cache what is deterministic. A meaningful share of production calls re-derive the same answer from the same input. Hash the resolved prompt and store the result.
Shorten context aggressively. Context is the largest controllable input cost in most systems. Retrieval that returns five precise passages beats retrieval that returns fifty adequate ones — on both cost and accuracy.
Constrain output shape. Structured output schemas cut output tokens substantially and remove parsing failures at the same time.
Where this leads
The end state is unremarkable and that is the point: a dashboard that shows spend per feature per day, an alert when a release moves the curve, and a routing layer that can be tuned without touching feature code. Cost becomes a number you manage rather than a surprise you absorb.