Features
Four capabilities you don't get from a plain LLM proxy.
Eval-driven smart routing
Route on evidence, not heuristics.
- A pluggable SmartScorer hook consumes your evaluation results — offline test-set scores plus live judge/human annotations — and returns a routing verdict per request.
- Cold-start behaviour is safe by default: unknown prompts go to the strong model. As evaluation coverage grows, weak-tier traffic ramps up automatically.
- Routing decisions surface in the response envelope so you can debug why a call went where it did.
9 built-in guardrails
Pre-call and post-call hooks that run inside the gateway.
- PII detectors (email, phone, SSN, credit card, IPv4) with two modes: block or redact.
- Blocklists, regex block, regex replace — deterministic, sub-millisecond.
- LLM-judge content filter that offloads soft policy checks to a cheap model.
- JSON-schema validation on responses — reject non-conforming outputs before they hit your app.
Semantic response cache
Two flavours in one column.
- Exact-match hits on prompt hash — near-zero-cost repeats for chatbots, agent loops, and popular queries.
- Similarity-match hits on embedding cosine distance — different wording, same intent, still a hit.
- Per-application TTL, cache mode, and similarity threshold. Guardrails re-run on cache hits so a tightened rule catches stale entries.
Multi-provider fan-out
One endpoint. Eight providers.
- Amazon Bedrock, OpenAI, Anthropic, Google Gemini, Qwen, DeepSeek, Zhipu, Moonshot — behind a single OpenAI-compatible endpoint.
- Fallback chains: primary provider throttled? The gateway retries against the next.
- Load balance across multiple deployments (e.g. two Bedrock regions) with round-robin or latency-aware policies.
- BYOK credentials, Fernet-encrypted at rest, optionally wrapped by KMS.
Try it in 30 seconds
The self-hosted image needs zero cloud credentials for a smoke test.