What is an AI gateway?
By Identra · Updated
An AI gateway is a control point between applications and AI services that routes requests and can enforce access, usage and content policies. It only protects the traffic and operations routed through it, so deployment coverage matters as much as its features.
How does an AI gateway work?
Your app calls the gateway instead of calling OpenAI or Anthropic directly. The gateway checks who is calling, applies policy and forwards allowed requests to an approved model. Depending on the product it can hold the provider keys, enforce rate and spend limits, inspect content and log requests. Azure API Management, Cloudflare AI Gateway, Kong AI Gateway and the open source LiteLLM proxy all sit in this category.
Products differ a lot. Some only handle model APIs. Others can also sit in front of agent tools. Microsoft's gateway documentation covers controls for both models and MCP servers. Check the routes and policies your gateway actually enforces.
Identity is the easy thing to get wrong. One shared API key tells the gateway which app called. It says nothing about which employee was behind the request. If policy depends on the person, pass user context through a channel the app cannot fake. And guard the keys themselves with ordinary API key security.
Why do teams put in an AI gateway?
One place to set which models are allowed, cap usage and change policy without touching every app. It also makes switching providers easier, though different models still behave differently and support different features.
Security teams get a record of which app called which model and whether policy allowed it. Be careful with full prompt logging. It builds a new store of sensitive data with its own access and retention problems. An AI audit trail usually needs the metadata far more than the content.
What does an AI gateway look like in practice?
Say a support team runs an internal assistant that summarizes tickets. The app fetches only tickets the employee may read and sends that text through the gateway to an approved model. A content rule blocks requests containing AWS keys or other secret formats the gateway recognizes. Logs record the app, the model and whether the request was allowed.
The gateway has no idea whether that employee should see that customer. The app decides that. If the assistant can also send email or update tickets, those actions need their own authorization where they execute.
Then test the ugly paths. A retry that falls back to calling the provider directly skips every policy. A fallback model may come with different data terms. A blocked request should fail cleanly without echoing the secret into an error message or debug log.
How is an AI gateway different from an AI firewall?
The names overlap and vendors use them loosely. A gateway leans toward routing and traffic management. An AI firewall leans toward inspecting content for security. Plenty of products do both. Compare what each one can inspect and block.
AI gateway
- Usual focus
- Routing, authentication, usage limits, logging
- Check
- Which model and tool calls actually pass through it
AI firewall
- Usual focus
- Content inspection and security decisions
- Check
- Supported content types and what happens on a block
Secure web gateway
- Usual focus
- General web access policy
- Check
- Visibility into encrypted traffic and in-app activity
What does an AI gateway miss?
Everything that does not go through it. An employee pasting a contract into ChatGPT on a personal login never touches your internal gateway. Neither does a desktop AI app or a model running locally in Ollama. Gateway logs are not a full picture of shadow AI.
Agents split the problem further. Claude Code can be pointed at a gateway for its model calls. Its file reads, shell commands and MCP calls still happen on the laptop. A gateway that supports tool traffic controls those calls only when they are routed through it.
Content inspection can catch some prompt injection. It cannot tell you whether an action the model asks for is allowed. OWASP's prevention guidance calls for layered controls, including limited tool privileges and human approval for destructive operations.
How do you deploy an AI gateway safely?
Map the traffic first. For each app, write down allowed models, allowed data, who may call and what happens when the gateway is down. Give every exception an owner and an end date. Pair the gateway with AI agent authorization and with controls in the browser, on the endpoint and on connected apps.
- Authenticate each app or workload, and stop callers from spoofing user context.
- Block direct provider access where you can. Check retries, failover and spare keys.
- Test inspection with real prompts, file uploads, streaming responses and tool messages. Write down what it cannot parse.
- Decide whether requests fail open or closed when a dependency breaks, then test it.
- Store as little content as you can and keep credentials out of logs.
How Identra thinks about it
An AI gateway sees the traffic routed through it. Identra looks at two places that traffic often skips: the browser and the device. In the browser it finds the AI apps people use and the accounts they sign in with. It can allow, redirect to the company AI workspace or block by account, and it checks prompts on the device before they are sent. On macOS and Windows devices, prompts to Claude Code and Codex are checked before they are sent, AI agent tool calls are checked against policy and every agent run is recorded.
Go deeper: AI security, built on identity
Frequently asked questions
Does an AI gateway cover every AI tool employees use?
No. Only what is routed through it. Third-party AI sites, desktop apps and local models need their own coverage.
Can an AI gateway stop sensitive data reaching a model?
If it has content inspection, it can block or redact the data types it supports. Routing, file formats and policy setup decide how much it actually catches.
Can coding agents use an AI gateway?
Some can be pointed at a custom model endpoint. That covers model calls. Local file access, shell commands and tool calls stay outside it.
Does an AI gateway replace app authorization?
No. Permission to call a model is not permission to read every record or run every action connected to it.
Should the gateway store every prompt and response?
Only with a clear reason and strong protection. Metadata and policy outcomes are enough for routine monitoring.