Detect the failure
The gateway reads provider results from governed requests. A 429 response or an exhausted-quota header pauses that target for a set period. Repeated upstream server failures open a short health circuit. A later success may clear the circuit. It does not erase a separate quota cooldown.
Keep every gateway instance in sync
Each tenant's cooldown and circuit records live in the shared control store. They contain operational state, not prompts or responses. Every gateway instance reads the same state, so the next request avoids a target that is already known to be unhealthy.
Select an approved fallback
First, the route excludes targets that are paused, cooling down, unhealthy, incompatible, over capacity, blocked by policy, or over budget. It then applies the configured priority, weight, or estimated-cost rule to the eligible list.
- Each target has a limit on attempts and timeouts.
- The request ledger records the skipped target, the fallback reason, and each provider attempt.
- An operator can drain a target and restore it later while keeping its configuration.
- If no valid alternative remains, the gateway returns an explicit error and makes no additional provider call.