Perspectives
You cannot cut a bill you cannot read
Caching and batching only help once you can measure.
By Ehsan Gazar, 16 September 2026
Your provider invoice arrives as one number. You know the total went up this month. You do not know which feature did it, which model, or which week.
Cheaper tokens do not touch that problem.
The bill with no line items
A provider bills the account, not the feature. One key, one total, one line on a statement. Everything your company built on top of that key arrives blended into a single figure.
So three questions have no answer. Which service spent the money. Which model it spent it on. Whether the spend was worth it.
Without the first two you cannot even ask the third. An expensive feature and a popular one look identical from the invoice, and the difference between them is the whole decision.
Asking people does not recover it either. The team that built the feature knows what it does, not what it costs, and by the time the invoice lands that work is three weeks behind them.
Optimisation without attribution is guessing with extra steps.
What one request has to say
Every response through our gateway carries what just happened, on the response itself. A request id. The model that actually answered, which is not always the one that was asked for. The prompt and completion token counts. How many of those prompt tokens were cached. The cost in dollars, to ten decimal places.
Those are not estimates. I generated the photograph at the top of this post through the same gateway while writing it. The response said req_mu3zdtuc7w7hjzy635, google/gemini-3.1-flash-image, and $0.0715239000, and the key it was charged to moved by exactly that.
Which model answered matters more than it sounds. A request can name a family rather than a version, or ask for whatever is cheapest that can do the job, and something has to resolve that. When a bill surprises you, the first question is which model actually ran, and the response answered it before anybody went looking.
Thirty days of request lines sit behind it, searchable by service, by level, by request id and by response time. One request can be followed across every service that touched it.
The unit is the request, not the month. That is the whole difference, and everything below depends on it.
The tokens you have already paid for
A long system prompt and a set of tool definitions get resent on every turn of a conversation. Providers charge less for a prefix they have already seen, because they have already done the work.
Here is the part worth telling you. Until 5 September 2026 this gateway charged every prompt token at the full input rate and read no cached count at all. Our pricing page said a token costs the provider's list price plus 5%, and on a cached prefix that claim was wrong by between four and ten times.
The fix was not to remember harder. Each vendor's published cached rate is stored as a multiple of its input rate, so a cached token lands on the cached list price plus 5% by construction. A model with no recorded cached rate keeps the full input rate rather than being given a discount nobody verified.
You can also see it working. The response tells you how many of your prompt tokens were a repeat, so the saving is a number you can watch rather than a claim we make.
A saving you cannot see is a saving you cannot prove.
One charge instead of ten thousand
Plenty of work does not need an answer this second. Embeddings for a corpus, classifying a backlog, regenerating summaries after a prompt change. Submitted as a batch and collected later, that work costs less.
The mechanism underneath is where the care goes.
In an ordinary batch every item is a real metered call. A key with a ten dollar cap that reaches it on item four thousand refuses item four thousand and one, and the remaining six thousand never run. The cap stops the spending partway through, which is the entire reason to set one.
A discounted batch is submitted in one piece and comes back as one charge. There is exactly one moment at which anything can be refused, and it is before submission.
That matters because the ordinary checks ask whether you have already spent your budget, not whether this would exceed it. Both readings are correct while the unit of charge is one call. Collapse ten thousand calls into one charge and a team with a ten dollar monthly cap and nothing spent yet would pass the check and be billed four hundred.
So a discounted batch is admitted only after the money is checked three times, at creation, before a single row is written. The discount is real. So is the brake it removes, and something has to replace it.
What cheaper tokens do not fix
None of this tells you whether the spend was worth it. Attribution says which feature spent the money. Only you can say whether that feature should have.
Caching helps a repeated prefix and nothing else. A workload that sends a different prompt every time saves nothing from it, and should be told so rather than sold a number it will never see.
Batching trades latency for price. If somebody is waiting for the answer, there is no saving there to take, and a vendor who implies otherwise is describing a different workload than yours.
Measure first. Every lever after that is arithmetic, and the only ones worth pulling are the ones you can watch move.