AI governance without storing every prompt

The short answer

You can govern AI use without turning the gateway into a prompt archive. For many cost, access, and policy questions, the useful record is what happened around the request rather than the full conversation.

AI Gateway HQ does not add prompt or response text to its durable request ledger by default. The ledger records operating evidence such as the workspace, workload, user or principal, route, provider, model, policy result, token use, cost, timing, and outcome.

What metadata can answer

That record can show which team made a request, whether the route and model were allowed, which budget paid for it, why a fallback occurred, and where a failure happened. It can also support spend allocation and policy review without asking an auditor to read employee conversations.

Tool names and types, MCP host names, decision codes, and final outcomes can be recorded as bounded metadata. Tool schemas, arguments, results, prompts, and model responses stay out of the durable request ledger by default.

Inspection is not the same as retention

Some controls must inspect content while a request is in flight. An authorized administrator can separately enable transient attack screening for hosted inference. The gateway uses the content to make that decision, but the setting does not turn prompt or response text into durable request evidence.

This distinction should be visible in policy and audit review. Teams should know whether content is merely processed for a live control, stored for a feature, or retained as an official record.

Check the full data path

A gateway setting does not control every system in the path. The calling application may log a request. A model provider may keep data under its account settings and contract. Prompt caches, MCP servers, tools, SIEM exports, support records, and customer-configured destinations may follow different rules.

Document each location, its purpose, owner, retention period, deletion process, and contract. Then verify the settings with a test request. Saying that the gateway does not retain prompt text is not the same as claiming zero retention from end to end.

• Confirm what the application logs before it calls the gateway.
• Review provider retention by account, project, endpoint, model, and feature.
• List every cache, tool, MCP server, export, and support path that can receive content.
• Test deletion and access controls instead of relying on a policy statement alone.