LiteLLM is the leading open-source LLM gateway: a proxy that gives you one unified API across 100+ model providers, with virtual keys, per-key budgets, and rate limits — free to self-host. Teams go looking for alternatives for a handful of recurring reasons: the operational burden of self-hosting a piece of critical-path infrastructure, budgets that stop at the API-key level, no way to intervene in a running agent session, observability that goes only as deep as a proxy log, and the need for enterprise support with someone on the other end.
This page walks through the real alternatives honestly — what each is good at, what it isn't — and then covers the layer most "gateway alternative" comparisons miss entirely: the spend control plane that sits above gateways, which may be what you were actually looking for.
What LiteLLM Does Well
Credit where it's due, because LiteLLM earned its position. It is genuinely open source and free to self-host, which matters enormously for teams with data-residency requirements or no budget line for middleware. Its provider coverage is the widest in the category — 100+ providers behind one OpenAI-compatible interface, so swapping gpt-4o for claude-sonnet is a config change, not a refactor. The community is large and active, which means fast support for new models and a deep well of answered questions. And the core governance primitives — virtual keys, team budgets, rate limits, spend logging — are real features, not toys.
If you are a small team that wants provider abstraction and basic budget hygiene at zero license cost, LiteLLM is a defensible default. Many of the teams reading this page should keep running it.
Why Teams Look for Alternatives
The reasons teams outgrow or route around LiteLLM are less about flaws and more about fit. These are the fit questions worth asking:
- Do you want to operate it? A self-hosted proxy on the critical path of every LLM call is infrastructure you now scale, upgrade, patch, and page someone about. Some teams happily own that; others discover they'd rather pay someone to.
- Is per-key budgeting enough? LiteLLM's budgets attach to virtual keys and teams. That caps aggregate spend, but it can't tell you which person, agent, or application behind a shared key spent the money, and finance will eventually ask.
- What happens mid-run? A budget that rejects the next request after a key is exhausted is not the same as enforcement inside a running agent session. A runaway loop can burn through a generous key budget before anyone looks up.
- How deep does your observability need to go? Proxy logs answer "what was called." Debugging quality regressions, tracing multi-step chains, and scoring outputs are a different discipline, and some alternatives lead with it.
- Do you need someone to call? Enterprise procurement often requires SLAs, support contracts, and a vendor relationship that a community Discord can't provide.
Different answers point at different tools. Here's the field.
The Alternatives
Portkey
Portkey is the most direct commercial substitute: an AI gateway with routing, fallbacks, caching, and virtual keys, plus a guardrails layer for enforcing request/response policies, available hosted or self-hosted. Its strengths are polish and the guardrails framework; its limits are that budget controls remain key- and workspace-scoped, and the deeper platform features sit behind paid tiers. Best for: teams that want LiteLLM's shape with a managed option, a guardrails layer, and a vendor behind it.
Helicone
Helicone is an observability-first gateway: it began as an LLM logging and analytics layer and grew gateway features around it, with an open-source core you can self-host. Its strength is depth of request-level visibility — logs, sessions, costs, prompt experiments; its limit is that it focuses on watching traffic rather than governing it, with controls that are coarser than its analytics. Best for: teams whose main gap is understanding what their LLM traffic is doing, not capping it.
Kong AI Gateway
Kong AI Gateway extends the well-established Kong API gateway with AI-specific plugins — multi-provider proxying, prompt guarding, token-based rate limiting — managed with the same tooling as the rest of your API estate. Its strength is exactly that consolidation; its limit is that it's a heavyweight adoption if you aren't already a Kong shop, and its controls are traffic-shaped, not spend-accountability-shaped. Best for: platform teams already running Kong who want LLM traffic governed like every other API.
Cloudflare AI Gateway
Cloudflare AI Gateway is the lowest-friction option on this list: change your base URL and you get caching, rate limiting, retries, and analytics at Cloudflare's edge, with a generous free tier. Its strength is that near-zero setup; its limits are that the controls are comparatively coarse, and it's a hosted edge service — your traffic metadata flows through Cloudflare, which rules it out for strict on-prem environments. Best for: teams that want basic caching and rate limiting today with minimal integration work.
TrueFoundry
TrueFoundry positions its AI gateway as one component of a broader ML/LLMOps platform — deployment, orchestration, and governance together, deployable in your own cloud. Its strength is that platform breadth; its limit is the same thing: you're evaluating a platform, not a drop-in proxy swap, and the gateway makes most sense if you want the rest of the platform too. Best for: teams shopping for an ML platform who want the gateway included rather than bolted on.
OpenRouter
OpenRouter is a hosted multi-model routing service: one API and one account for hundreds of models across providers, with per-key credit limits and pay-as-you-go billing. Its strength is sheer convenience — no infrastructure at all, instant access to new models; its limits are that it's hosted-only, sits between you and the providers commercially as well as technically, and offers little of the enterprise governance surface. Best for: product teams and individual developers who want frictionless multi-model access, not an enterprise control layer.
The Layer Above Gateways
Here is the thing every tool above has in common, LiteLLM included: they watch and cap traffic at the proxy. They differ in hosting model, depth of analytics, and how the caps are expressed — but the job is the same job. Route the request, log it, reject it if a key is over budget. That's the valve.
A spend control plane does a different job. It also decides:
- Who spent it. Not "which API key" — which person, which agent, which application. Per-entity attribution means every cent has a name, so chargeback, showback, and the end-of-quarter finance conversation stop being archaeology.
- What happens mid-run. When an agent session starts compounding — retry loops, spawned sub-agents, a task someone walked away from — a control plane evaluates policy between rounds within the session and can stop it with a kill switch at call 12, not discover it at call 400. A per-key budget cannot do this; by the time the key is exhausted, the damage is the size of the budget.
- Whether an optimization actually works. Model routing, caching, prompt changes — every gateway can apply them, but none tells you what they'll do to your quality bar. A control plane backtests recommendations on your own historical workloads, measuring cost and quality impact before you commit.
This is the framework we call the AI Spend Control Loop — observe, attribute, forecast, stress-test, verify, enforce — and it's the layer Oolyx occupies. To borrow our own one-liner about gateways: those watch and cap traffic; we enforce and decide — mid-round kill switch, quota backtesting, evals on your own workloads, pure on-prem.
Two honest clarifications. First, Oolyx is not a gateway replacement — it's complementary to (or a layer above) this list. It can sit alongside a gateway like LiteLLM: the gateway keeps doing provider routing and API unification, and Oolyx adds attribution, mid-run enforcement, and verified savings. Second, in-path enforcement is not free: Oolyx adds 10.4 ms of median overhead — about 0.4% of a typical LLM call. It runs on-prem with no telemetry, which is the deployment shape this whole decision layer needs if your traffic can't leave your network. For the broader discipline this belongs to, see our primer on FinOps for AI.
Comparison Table
Based on our review of each vendor's public docs (September 2026). "—" means we couldn't verify the capability either way — check the vendor's current documentation.
| Tool | Type | Hosting model | Budget granularity | Mid-run enforcement | Verified savings / backtesting |
|---|---|---|---|---|---|
| LiteLLM | Open-source LLM gateway | Self-hosted (paid enterprise options) | Per virtual key, team, model | No — per-key budgets and rate limits | — |
| Portkey | AI gateway + guardrails | Hosted or self-hosted | Per key / workspace | No — per-request guardrails, not session-level | — |
| Helicone | Observability-first gateway | Hosted or self-hosted (open source) | Per key | No | — |
| Kong AI Gateway | API gateway with AI plugins | Self-hosted or Konnect (hosted control plane) | Token/request rate limits per consumer | No | — |
| Cloudflare AI Gateway | Edge AI gateway | Hosted (Cloudflare edge) | Coarse — per-gateway limits | No | — |
| TrueFoundry | Gateway within an ML platform | Self-hosted (your cloud) or hosted | Per key / team, per public docs | No | — |
| OpenRouter | Hosted multi-model router | Hosted only | Per-key credit limits | No | — |
| Oolyx | AI spend control plane | On-prem, no telemetry | Per person, agent, and application | Yes — mid-round kill switch | Yes — backtested on your own workloads |
In our review of public docs, Oolyx was the only tool in this set with both mid-run enforcement and backtested recommendations — which is less a scoreboard claim than a category boundary. The seven tools above it are competing to be the best valve. The bottom row is a different layer, and it works with a valve, not instead of one.
Frequently Asked Questions
What is the best LiteLLM alternative?
It depends on what pulled you away from LiteLLM. Want a managed gateway with guardrails and a vendor relationship? Portkey is the most direct substitute. Is observability the gap? Helicone leads with it. Already running Kong or Cloudflare? Their AI gateways extend infrastructure you have. Just want hosted multi-model access with zero ops? OpenRouter. And if the gap is enforcement and accountability — stopping a runaway agent mid-run, knowing who spent what, proving an optimization works — that's a control plane question rather than a gateway question, and it's the layer Oolyx occupies.
Is LiteLLM free?
Yes — the core project is open source and free to self-host, including the proxy with virtual keys, budgets, and rate limits. The company behind it offers paid enterprise tiers with additional features and support. The real cost of the free version is operational: you run, scale, and secure the proxy yourself.
Can I use Oolyx with LiteLLM?
Yes. Oolyx is complementary: it can sit alongside or above a gateway, so LiteLLM keeps handling provider routing and API unification while Oolyx adds per-entity attribution, mid-run kill-switch enforcement, and backtested recommendations. Both deploy on-prem, so nothing about the combination requires traffic to leave your network.
What's the difference between an LLM gateway and an AI spend control plane?
A gateway watches and caps traffic: it routes requests, logs usage, and rejects calls once a key's budget or rate limit is hit. A control plane also decides — it attributes every dollar to a person, agent, or application; intervenes mid-run when a session runs away; and verifies optimizations by backtesting them on your own workloads before you commit. The AI Spend Control Loop lays out the full framework.