AI gateway cost control: stop surprise spend before dispatch

Start with the decision the gateway must make

An AI cost dashboard can show that a workload spent too much. It cannot recover the money. A pre-dispatch control has a narrower job: identify the workload, estimate the maximum eligible exposure, test the applicable limits, and either reserve the amount or stop before provider credentials are used.

Treat the gateway estimate as an operating control, not the provider's final invoice. Model prices, caching, service tiers, retries, provider rounding, taxes, and incomplete responses can make modeled and billed totals differ. Keep the provider's own quota or spend boundary enabled as an independent backstop.

1. Give every workload one owner and one key

Create a separate workload key for each application, environment, agent, or cost owner. Do not share one unrestricted key across unrelated teams. Shared credentials make attribution, rate control, incident response, and revocation ambiguous.

Bind the key to a stable route and a plain-language business purpose. A finance review should be able to distinguish a customer-support assistant from a coding workflow without reading a prompt or learning a provider's model identifiers.

2. Bound the request before pricing it

Set the smallest defensible output cap and reject inputs that exceed the route's approved context boundary. Providers may reserve rate-limit capacity using the input plus the requested maximum output before later adjusting to actual usage. An unnecessarily large output cap can therefore reduce concurrency even when the final answer is short.

Use separate request-rate, token, concurrency, and cost limits. A dollar ceiling does not stop a burst from exhausting provider throughput, and a token quota does not necessarily equal the amount eventually billed.

3. Reserve conservative exposure atomically

For a strict budget, estimate the request against the eligible model rate and output cap, then reserve that exposure in the same atomic decision that authorizes dispatch. Concurrent requests must not each observe the same remaining balance and all spend it.

After a supported response reports usage, settle the reservation to the observed token amount. If the provider outcome is ambiguous, retain the conservative reserve until the charge boundary can be reconciled; releasing it optimistically can authorize spend that the provider still bills.

4. Decide continuity before a limit is reached

Choose one explicit behavior for each workload: stop, use an already-approved lower-cost target, or continue from separately prepaid credit. A fallback remains subject to the same capability, data, security, health, and budget requirements; a cheaper but unsuitable model is not an eligible alternative.

For managed usage, keep automatic reload off until an owner chooses the trigger balance, reload amount, and hard monthly reload cap. A pending or failed payment must never create spendable credit, and reaching the cap must stop another reload rather than silently turning into overage.

5. Use caching only where the workload permits it

Repeated stable context can reduce provider input cost and latency when the selected model supports prompt caching. Cache reads and writes may have different prices, minimum context sizes, and retention behavior, so calculate the break-even point for the actual request pattern.

Do not enable a cache solely because it is cheaper. Confirm workload isolation, content sensitivity, data-retention requirements, tool behavior, and invalidation first. A cache hit still belongs to the workload's gateway activity and platform-fee accounting even when it avoids another provider call.

6. Reconcile the control ledger with the provider bill

Compare the provider's financial cost source with gateway evidence for the same provider account and UTC period. Separate exact provider-reported usage from modeled or conservatively settled events, and disclose retained-event coverage before interpreting a variance.

OpenAI, for example, documents that granular Usage API output may differ from financial costs and directs financial reconciliation to its Costs endpoint or billing dashboard. The general lesson is provider-independent: operational token evidence is not automatically an invoice.

• Investigate unrecognized provider spend that has no gateway workload attribution.
• Investigate gateway-modeled spend that is absent from the provider financial total.
• Keep contract discounts, service tiers, caching, taxes, and credits visible as possible variance sources.
• Do not call an estimated model substitution a saving until a comparable quality and cost experiment supports it.

7. Review exceptions every week

Start with exhausted or nearly exhausted budgets, concentrated provider exposure, repeated fallback, high failure cost, uncapped workloads, and spend that has no named owner or business result. Review the exception and required decision before reviewing a leaderboard of token totals.

A useful decision record states who owns the change, what control or route will change, the expected effect, the comparison window, and the acceptance condition. Keep policy approval separate from the automated recommendation.

A 30-minute first implementation

Begin with one non-sensitive workload. Issue one workload key, create one route with an explicit primary target, add a compatible fallback only if its behavior has been tested, set conservative request and monthly limits, and run synthetic policy cases before live traffic.

Then send a small governed request, locate its reason-coded evidence, and reconcile its reported usage against the provider account. Expand only after the owner can explain the stop behavior and the cost basis without relying on the implementer.