You don't want your whole agent stack hardcoded to one model provider's pricing and uptime. A gateway sits between your code and every provider, giving you one API to route, cache, and observe calls across all of them.
Core concepts
- OpenRouter: The most widely used unified API/model marketplace — one endpoint, dozens of models. OpenRouter Docs
- Portkey & Helicone: AI gateway + observability in one layer — routing plus cost and latency tracking out of the box. Portkey Docs · Helicone Docs
- LiteLLM: The open-source unified gateway most teams self-host when they want full control. LiteLLM Docs
- Vercel AI Gateway & Cloudflare AI Gateway: Platform-native gateways with routing, caching, and analytics built in if you're already deployed on that platform — pair them with Vercel AI SDK or Cloudflare Workers AI so your app code stays on one streaming/tool-calling API while the gateway handles provider keys, fallbacks, and usage dashboards. Vercel AI Gateway · Cloudflare AI Gateway
- When to add a gateway: Not needed for a weekend project — becomes worth it the moment you're running the same agent against more than one provider or need centralized cost tracking.
What Is It?
LLM gateways abstract provider APIs behind one interface.
They enable fallbacks, budget caps, and centralized logging.
Platform clouds (Azure, Bedrock) also offer gateway-like model catalogs.
Why Should I Care?
Agents multiply API calls — gateways are where finance and SRE teams set controls.
Switching models for evals is a config change, not a rewrite.
Real World Analogy
A gateway is a hotel concierge — you ask once, they figure out which vendor room service to call and bring you one bill.
Visual Mental Model
your app -> gateway -> provider A/B/C
-> cache / logs / budgetsBeginner Explanation
Gateways like OpenRouter, LiteLLM, Portkey, and Helicone sit between your agent and providers for routing, caching, observability, and key management.
Smallest Possible Example
# LiteLLM / OpenRouter unify routes
route = {"model": "anthropic/claude-sonnet", "fallback": "gpt-4.1-mini"}Common Mistakes
❌ Logging prompts with PII to third-party gateways unchecked.
Redact or hash sensitive fields; read retention policies.
❌ Infinite fallback chains.
Each fallback costs latency — define sane stop conditions.
❌ Treating gateway as auth boundary alone.
Still validate outputs and tool args in your app.
Real Project Example
LiteLLM proxy is common in self-hosted agent platforms.
Portkey markets guardrails plus routing for enterprises.
Build This Mini Project
Configure two models through one gateway SDK and toggle between them with an env var; log latency for each.
Interview Question
What belongs in a gateway vs your agent code?
Gateway: routing, keys, rate limits, caching, provider-normalized logging. Agent code: business logic, tool auth, validation, and user permissions.
Key Takeaways
- One interface, many providers.
- Use for fallbacks and spend caps.
- Mind PII in logs.
- Keep validation in app layer.
Resources
YouTube learning
Core concepts from video walkthroughs:
The 60-second story
One power company is simple until they have an outage during your demo, or they change the night rate and your bill learns to fly. A gateway is the building electrician who sits between your workshop and every utility. Your code asks the electrician. The electrician decides who actually supplies this call, logs the meter, and can fail over when a utility coughs.
OpenRouter is the market where one outlet reaches a lot of models. You trade a bit of purity for a single bill and a single shape. Portkey and Helicone are the electrician who also hands you a clipboard: cost, latency, which model, which user. You want that clipboard the week a loop runs hot. LiteLLM is the open-source electrician you host yourself when the clipboard must live in your building.
Vercel and Cloudflare ship electricians that already know their neighborhoods. If the app is on that platform, the gateway is one less vendor to introduce at the architecture review. Your SDK still speaks one streaming, tool-calling dialect. The gateway holds the keys and the fallbacks.
You do not hire an electrician for a lamp in a bedroom. You hire them when two utilities are in play, or when finance asks “which feature spent this” and your logs say “the AI did it.”
Map the building.
- Your app is the workshop.
- A provider is a utility.
- The gateway is the electrician and the meter.
- OpenRouter is a shared market outlet.
- Portkey or Helicone is the electrician with a clipboard.
- LiteLLM is the electrician you employ.
- A platform gateway is the electrician who came with the building.
app -> gateway -> provider A or provider B
logs live on the gateway, not in a print statementThat arrow is the design. If the app imports three vendor SDKs and a pile of if provider, you built a gateway badly, inside the product.
Gateways exist because provider choice became a runtime decision: price, outage, data residency, a model that is better at code on Tuesdays. They are a layer, which means they can also be a new outage. Add one when the pain is real. Weekend projects can call a provider directly and stay honest.
Mental model: one outlet in the workshop, many utilities behind the wall. Your code should not know which truck delivered the electron.
You have the page if you can say when a gateway is extra furniture, and what you would look up on it during an incident: route, cost, latency, fallback.
