How to Choose an AI Gateway for Multiple LLM Providers
Quick answer: If you call two or more LLM providers, a gateway gives you one API, fallbacks when a provider fails, and one place to cap spend. Choose by where it has to run, how it charges, how it handles failures, and what it can tell you about cost. LiteLLM if you need it on your own servers, OpenRouter for one bill across many providers, Cloudflare AI Gateway if you already run on Cloudflare. None of them tells you which product feature or customer spent the money; that's a separate layer.
The first provider is easy. The second is where the code starts to sprawl: two SDKs, two sets of keys, two error formats, and a retry loop someone wrote late at night. A gateway puts all of that behind one endpoint. Picking one comes down to four questions, and most comparisons skip the last one.
This is part of our guide to LLM and AI cost optimization.
Do you need a gateway at all?
Probably not if you use one provider and one or two models. The provider SDK already handles retries, and every extra hop is one more thing that can fail.
You probably do if:
- you call more than one provider, or want to switch providers without a rewrite
- you need a fallback when a provider returns errors
- several services or teams share provider keys and need separate limits
- you want caching or rate limits in one place instead of in every service
The cost of rewriting an app for a new provider is the usual reason teams add a gateway later than they should.
1. Where does it run?
Hosted gateways such as OpenRouter, Cloudflare AI Gateway and Portkey mean nothing to operate, but every request passes through someone else's infrastructure. A self-hosted proxy such as LiteLLM runs on your servers: you own the uptime and the upgrades, and request data stays on your network. If compliance decides this for you, the list gets short quickly.
2. How does it charge?
Three models are common:
- A fee on credits. OpenRouter charges 5.5% ($0.80 minimum) when you buy credits and says there's no markup on inference prices.
- A plan with a log allowance. Portkey's free Developer plan lists 10,000 logs a month. Production is $49 a month for 100,000, then $9 per extra 100,000.
- Your own servers. LiteLLM costs whatever it costs to run, plus an enterprise license for features such as spend by tag.
Work it out at your real volume. A percentage fee is small on $200 a month of model spend and noticeable on $20,000.
3. What happens when a provider fails?
Fallback is the main reason to have a gateway, so test it before you commit: force an error from your primary provider and watch what the gateway does. OpenRouter documents automatic fallback to the next provider when one errors. Cloudflare AI Gateway lists retries and model fallback.
Check whether a fallback switches to a different model. A cheaper or weaker model quietly answering in place of your usual one changes both quality and cost, and you want to know when it happens.
4. What does it tell you about cost?
Every gateway reports spend. They differ in what they group it by:
| GATEWAY | SPEND GROUPED BY |
|---|---|
| LiteLLM | Key, user, team; tags on enterprise |
| Portkey | Request, model, provider, key |
| Cloudflare AI Gateway | Analytics that include cost |
| OpenRouter | Not covered here |
From each product's own docs, 16 October 2026.
Spend per key answers "which service spent it". It doesn't answer "which feature in the product" or "which customer", unless you create a key per feature and per customer, which nobody keeps up for long.
Check one more thing: what the gateway does with a model it has no price for. Portkey's docs say requests to models without pricing show $0.00, which looks exactly like a free request in a report.
Caching: ask which kind
Cloudflare AI Gateway can serve repeated requests from its cache. Before you count on savings, ask each vendor whether its cache is exact-match (the same request, character for character) or semantic (a similar request), and how long entries live. Exact-match caching helps with retries and repeated calls; it does little for open-ended chat.
Provider-side prompt caching is a separate discount that works with or without a gateway: see how prompt caching cuts LLM cost.
Which one, by situation
- It has to run on your own servers: LiteLLM.
- You want many models without many provider accounts: OpenRouter.
- Your app already runs on Cloudflare: Cloudflare AI Gateway.
- You want spend caps enforced in the request path, and an enterprise-focused vendor suits you: Portkey, now part of Palo Alto Networks.
- You're on Helicone today: it has been in maintenance mode since March 2026. It still works, but plan the move. Here's a Helicone alternative for LLM cost tracking.
For more options, see these Portkey alternatives for small teams, or the wider list of LLM cost tracking tools, compared.
The layer a gateway doesn't cover
A gateway knows the request. It doesn't know that the request came from your summarizer, for customer acme-corp, who pays you $29 a month. That takes a label on each request and something that adds the labels up.
LLMtrack does AI cost tracking per feature and customer from the traces your gateway already exports. One URL in the gateway's settings, no SDK:
Each request is then priced from the AI API pricing sheet. A model with no published price is flagged as unpriced, never shown as $0.
FAQ
What is an AI gateway?
A service between your app and the LLM providers. It gives you one API for several providers, with routing and fallbacks, and usually caching, rate limits and spend reporting.
Does a gateway add latency?
It adds a network hop. Measure it on your own traffic, from the regions your users are in, before you roll it out everywhere.
Can I use two gateways?
You can, but it doubles the places a request can fail. Most teams run one gateway and add a cost tracker beside it.
Is OpenRouter a gateway?
It's a hosted router: one API and one bill across many providers, with automatic fallback when a provider returns an error.
Keep your gateway, and see which feature and which customer spend the money. The free plan includes every feature: start free.
Related guides
- Portkey alternatives for small teams
- LLM cost tracking tools, compared
- Helicone alternative for LLM cost tracking
Last updated 16 October 2026. Gateway facts were checked on each vendor's own pages that day and change often.