Model Router Gateway: the smallest model that passes
A gateway that routes each task to the cheapest model passing your eval — frontier only for the hard step. Demo reference architecture.
- Industry
- Cross-industry / AI operations
- Build, estimated
- 3–5 weeks
At a glance
- It starts with
- Task comes in
- The machine handles
- 8 of 8 steps
- A person keeps
- Nothing — it runs end to end
- Bottom line
- smallest modelthat passes the test
The problem
Most automations send every step to the same expensive large model — even the trivial "is this an invoice? yes/no." That's a truck for the bread run.
This demo shows the opposite: a gateway that decides per step which model is enough — and calls the frontier only when the task genuinely needs it.
The industry is shifting from "more parameters" to "intelligence per euro." A 7B SLM is roughly 10–30× cheaper than a 70–175B model (NVIDIA, 2025) and is enough for the repetitive routine tasks that make up most of agent work — classify, extract, route, format. Meanwhile the price for a fixed capability level is falling dramatically (LLMflation: roughly 10×/year).
The catch: most mid-market companies don't use this. They have (a) a model hard-wired — lock-in, no switching when prices fall, (b) no eval to know whether a smaller model would do, and (c) no visibility into per-step cost. The result: they pay frontier prices for SLM work.
Honest limit: for hard, broad tasks (open-ended reasoning) the large model remains superior — inference compute substitutes for parameters only partially. The routing must respect that boundary, not ignore it.
The process, step by step
Scroll through. The diagram stays put and highlights the step you are on.
- 01
Task comes in
step in the workflow
A step in your workflow needs a model — classify, extract, summarize, or a difficult synthesis. Instead of blindly calling the large model, the task goes to the router.
- 02
Classify & eval
type · difficulty
The router determines the task's type and difficulty and checks it against the stored eval — the small, honest test per task type.
- 03
Model router
OpenRouter · LiteLLM
The model-agnostic gateway (OpenRouter / LiteLLM) abstracts over 400+ models. The provider is configuration, not hard-wired code — you switch when prices fall.
- 04Branch
SLM enough?
The decision: is a small model (SLM) enough for this task — does it pass the eval — or does it need the frontier?
- 05
Run SLM
cheap · fast
Routine tasks (the bulk) run on a cheap, fast SLM. Roughly 10–30× cheaper than a large model and entirely sufficient for classify, extract, route.
- 06
Frontier model
escalation · hard task
Only the demonstrably hard cases escalate to a frontier model. You pay the top price deliberately — not blanket for every call.
- 07
Log cost & eval
per step
Every call is logged with model, cost, and eval result. You see spend per step, per model, per day — no surprise on the monthly bill.
- 08
Result + cost view
back into the workflow
The result goes back into the workflow — produced by the smallest model that passes the task, with full cost visibility.
What changes
Possible setup, not a packaged product
The figures are modelled target values for a possible setup, derived from industry benchmarks and our own tests. They do not come from a customer project; real values depend on your business. We do not sell this setup as a finished product: we measure where your bottleneck is, then build with your team. For systems that require certification (e.g. HIS, gematik, DATEV-certified) we work with specialised partners.
Before vs. After
| Aspect | Before | After |
|---|---|---|
| Model choice | Before: one large model for everything | After: routed to fit per step |
| Routine tasks | Before: frontier price | After: SLM price |
| Hard tasks | Before: same model as routine | After: targeted frontier escalation |
| Provider switch | Before: code rebuild | After: configuration |
| Cost visibility | Before: lump bill at month-end | After: cost per step, live |
Technology
Technology Stack
Integrations
Systems this setup can connect to
- n8n (self-hosted)Orchestration
- Central engine: classification, routing rules, escalation, and logging
- OpenRouter / LiteLLMModel abstraction
- One API over 400+ models; switch providers by configuration
- SLM + frontier modelsInference
- Small models for routine, frontier on demand for hard tasks
- Eval suiteQuality control
- Task-specific tests decide which model gets passed through
- PostgreSQLCost/eval log
- Record of all calls with model, cost, and test result
- GrafanaCost dashboard
- Spend per step, model, and day — visible live
Security & Compliance
What the setup takes care of
- Data sovereignty through self-hosting
- Gateway, evals, and logs run self-hosted. With LiteLLM even the model abstraction stays in-house — no obligation to send prompts through someone else's router.
- No lock-in by design
- Models and providers sit behind an abstraction and are swappable by configuration. The most expensive lock-in is the one you write into your code — this setup deliberately avoids it.
- Auditable cost & eval log
- Every call is logged with model, cost, and eval result — traceable which model handled which task with what outcome.
- No quality promise without evals
- The routing is only as good as the tests behind it. Framed honestly: the evals are the core — without them, "smaller model" is a gamble; with them, a measurement.
Frequently Asked Questions
What would this look like at your company?
This workflow is a demo. Whether it pays off for you depends on where things get stuck at your company. The method page shows how to find out. To place a workflow of your own, use the free assessment. No call needed.
Related Showcases
Agent Permission Gateway: AI Agents on Least Privilege
A control layer between AI agents and their tools: narrow rights, short-lived tokens, approval gate, audit log. Demo architecture, not a product.
EU AI Act Cockpit: Prove Obligations Automatically
A central AI registry, an immutable audit trail, a human-in-the-loop gate and Article 50 transparency notices. Demo reference architecture.