Begin with the boundary you are buying
An enterprise AI gateway sits between applications and model providers. Its job is not merely to relay an API call. It must identify the caller, apply the current policy and budget, select only an eligible destination, and leave enough evidence for another operator to explain the result.
Write down the required deployment boundary, data classes, identity source, provider accounts, workloads, continuity target, and financial owner before comparing products. A polished dashboard cannot compensate for an undefined operating boundary.
1. Prove the application can change one endpoint
Connect an OpenAI- or Anthropic-compatible test client without rewriting its business logic. Confirm the gateway exposes only the route aliases approved for that workload and that the application never receives the underlying provider credential.
2. Prove workload identity is separate from provider identity
Issue a unique, revocable identity for one workload and environment. Verify that it cannot use another route, tenant, or environment, and that rotating the provider credential does not require distributing a new provider secret to applications.
3. Prove routing is constrained—not improvised
Create an alias with an explicit primary target and a compatible secondary target. Test model capability, context, tool, streaming, data-class, policy, capacity, and budget requirements. A routing strategy should optimize only among destinations that already passed those gates.
4. Prove provider failure does not become policy drift
Simulate rate limiting, timeouts, repeated provider errors, quota exhaustion, and an operator drain. Confirm health state is shared across gateway instances, retries are bounded, the approved fallback is explainable, and no eligible target returns a clear failure instead of an unapproved substitution.
5. Prove the budget stops spend before dispatch
Set a small hard limit, send concurrent requests near the boundary, and verify conservative exposure is reserved atomically before provider credentials are used. Test exhausted credit, a failed reload, a monthly reload ceiling, and an ambiguous upstream result. A notification after overspend is not a hard budget.
6. Prove cost evidence can be reconciled
Check the model, region, input, output, caching, service tier, and price version recorded for each request. Compare a bounded period with the provider's financial cost source and label modeled, provider-reported, missing, or truncated evidence. Do not treat an estimate as an invoice or a modeled substitution as realized savings.
7. Prove policy safely progresses from observation to enforcement
Run representative allow, deny, redact, and redirect cases in simulation. Confirm organization posture can cap a policy's effective mode, version conflicts stop stale changes, and enforcement cannot send traffic to a route that was not already approved.
8. Prove content inspection has a clear retention boundary
Test prompt-injection and tool-boundary signals with synthetic content. Identify exactly what is inspected, sent to another service, retained, encrypted, exported, and deleted. Verify prompt and response storage is off unless an authorized owner makes a separate, auditable choice.
9. Prove the audit trail answers an incident question
Choose one request and reconstruct who called, which configuration version applied, what decision was made, which provider attempts occurred, what usage and cost basis were observed, and whether an administrator changed a control. Verify tamper evidence and export boundaries without requiring prompt or response bodies.
10. Prove enterprise access works with existing administration
Test least-privilege roles, separation between customer and staff identity, SSO metadata rotation, SCIM lifecycle, break-glass handling, session expiration, recovery, and immediate revocation. A sysadmin should be able to determine who can change money, policy, credentials, and reporting without reading source code.
11. Prove tenant and deployment boundaries
Attempt cross-tenant identifiers, keys, routes, exports, support access, caches, stored context, and reporting links. For hosted and customer-deployed options, document who owns the cloud account, encryption keys, network path, logs, upgrades, backups, provider terms, quotas, and incident response.
12. Prove the operating claims on the exact release
Ask for evidence tied to the deployed revision: source and dependency checks, infrastructure plan, signed artifacts, configuration inventory, external security checks, authentication journeys, provider canaries, billing lifecycle, backup or restore evidence, and known exceptions. Internal automation is useful evidence; it is not an external certification or a human penetration test.
Use one decision sheet for every candidate
For each test, record required, passed, partially passed, failed, or not evaluated; attach the observed evidence; name the exception owner; and set a decision date. Keep commercial questions beside technical ones: implementation effort, hosted and private-deployment responsibility, metering basis, minimum commitment, support boundary, data terms, and exit path.
AI Gateway HQ exposes a no-card Test Lab for the first comparison and a free workspace for saved controls. Use fictional content first. Move one non-sensitive workload only after the evaluator can explain the route, stop condition, recovery behavior, cost basis, and retained evidence.
Sources and attribution
NIST describes cross-sector generative-AI risk-management considerations and actions for governance, content provenance, pre-deployment testing, and incident disclosure.
OWASP organizes recurring application risks including prompt injection, sensitive-information disclosure, improper output handling, excessive agency, and unbounded consumption.
AWS applies operational excellence, security, reliability, performance efficiency, cost optimization, and sustainability questions to generative-AI workloads.