A gateway, not an SDK: the pattern for routing LLM calls
Wiring a provider SDK into every call site hardcodes model choice. Routing through a gateway instead makes it configuration you can change in one place.
- Calling a provider SDK directly bakes model choice, retries, and token accounting into application code.
- A thin gateway that every model call goes through turns model selection into configuration, not a code change.
- It enables per-tenant model routing and per-step cost attribution, not just a single monthly total.
- Build it once you have a second model to route between, not in anticipation of one.
Most LLM features start with a provider SDK imported directly into the service that needs it. That is the right call for the first version. It becomes the wrong call at roughly the point where you have more than one model, more than one tenant, or a finance conversation about token spend.
What direct SDK calls actually couple
The import is not the problem. The problem is everything that accretes around it: retry policy, fallback, token accounting, prompt-cache handling, and the model name itself all end up as application code, duplicated per call site with small inconsistencies.
Then a model is deprecated on a date you do not control, and the change is a codemod across every service instead of a config edit.
What a gateway is and is not
A gateway here means a thin internal layer that every model call goes through. Not a product, not a proxy service you operate separately unless you need to - a library boundary is often enough. It owns four things:
- Model selection, resolved from configuration rather than a literal in code
- Fallback order when a provider degrades or rate-limits
- Token and cost accounting, attributed to a caller rather than a total
- Cache strategy, so the split between cacheable instruction and per-request context is decided once
The application asks for a capability - summarise, classify, plan - and the gateway decides which model serves it. That inversion is the whole point.
Where it earns its keep
Per-tenant routing becomes possible: one brand on a frontier model, another on something cheaper, with no branching in the feature code. Cost attribution becomes per-node rather than a single monthly number, which is the difference between knowing you spend too much and knowing which step is doing it.
If you cannot answer which agent step costs the most, you do not have a cost problem yet - you have an attribution problem.
The honest cost is one more indirection to debug through, and a real risk of over-abstracting before you have two models to route between. Build it when the second model arrives, not in anticipation of it.