Start with independent supply
Add at least two approved route targets whose credentials and provider paths are meaningfully independent. A second model on the same unavailable account may not provide useful failover.
Approve every model first
Name each provider account, model, priority, attempt limit, and timeout explicitly. The gateway never invents an unapproved provider or silently substitutes a model outside the released route.
Narrow before selection
For every request, the gateway removes targets that fail capability, region, policy, budget, credential, health, cooldown, or observed capacity checks before any routing strategy chooses a destination.
Open the circuit on failure
Provider errors update the target's health circuit and retry timing. An open circuit keeps new requests away until its bounded recovery window permits another controlled attempt.
Honor cooldown and capacity
Rate-limit and quota headers create bounded cooldown or capacity windows when supported. Those observations remove the affected target without disabling healthy alternatives in the same released route.
Preserve controls during fallback
A fallback does not bypass policy, guardrails, token limits, or the hard budget. The alternate target is evaluated under the same request identity and current control configuration.
Stop when nothing qualifies
If every approved target is unhealthy, cooling down, over capacity, incompatible, or otherwise ineligible, the gateway returns a reason-coded failure instead of weakening the boundary.
Review every attempt
After a request, review provider attempts, target exclusions, fallback reason, latency, usage, and final outcome. Test the scenario in the no-provider-call lab before expanding production traffic.