[HN Gopher] Show HN: Arch-Router - 1.5B model for LLM routing by...
___________________________________________________________________
Show HN: Arch-Router - 1.5B model for LLM routing by preferences,
not benchmarks
Hi HN -- we're the team behind Arch
(https://github.com/katanemo/archgw), an open-source proxy for LLMs
written in Rust. Today we're releasing Arch-Router
(https://huggingface.co/katanemo/Arch-Router-1.5B), a 1.5B router
model for preference-based routing, now integrated into the proxy.
As teams integrate multiple LLMs - each with different strengths,
styles, or cost/latency profiles -- routing the right prompt to the
right model becomes a critical part of the application design. But
it's still an open problem. Most routing systems fall into two
camps: - Embedding-based routers use intent classifiers -- label a
prompt as "support," "SQL," or "math," then route to a matching
model. This works for simple tasks but breaks down in real
conversations. Users shift topics mid-conversation, task boundaries
blur, and product changes require retraining classifiers. -
Performance-based routers pick models based on benchmarks like MMLU
or MT-Bench, or based on latency or cost curves. But benchmarks
often miss what matters in production: domain-specific quality or
subjective preferences like "Will legal accept this clause?" Arch-
Router takes a different approach: route by preferences written in
plain language. You write rules like "contract clauses - GPT-4o" or
"quick travel tips - Gemini Flash." The router maps the prompt (and
conversation context) to those rules using a lightweight 1.5B
autoregressive model. No retraining, no fragile if/else chains. We
built this with input from teams at Twilio and Atlassian. It
handles intent drift, supports multi-turn conversations, and lets
you swap in or out models with a one-line change to the routing
policy. Full details are in our paper
(https://arxiv.org/abs/2506.16655), but here's a snapshot: Specs:
- 1.5B params -- runs on a single GPU (or CPU for testing) - No
retraining needed -- point it at any mix of LLMs - Cost and
latency aware -- route heavy tasks to expensive models, light tasks
to faster/cheaper ones - Outperforms larger closed models on our
conversational routing benchmarks (details in the paper) Links: -
Arch Proxy (open source): https://github.com/katanemo/archgw -
Model + code: https://huggingface.co/katanemo/Arch-Router-1.5B -
Paper: https://arxiv.org/abs/2506.16655
Author : adilhafeez
Score : 42 points
Date : 2025-07-01 17:13 UTC (5 hours ago)
| sparacha wrote:
| Hi HN! I am one of the co-authors of the paper. If there are any
| questions about our approach, I would love to answer them.
| tmaly wrote:
| do you think it would be possible to quantize this model and
| still get good results?
| sparacha wrote:
| yes - we have already published a quantized version here:
| https://huggingface.co/katanemo/Arch-Router-1.5B.gguf. The
| performance difference with a quant version is negligible. I'll
| run another analysis and update the thread shortly
| sparacha wrote:
| Overall performance degrades from 93.17 -> 92.99 with a
| quantized version
| jedisct1 wrote:
| I tried to use it to rate the difficulty level of coding tasks
| (for InferSwitch, an LLM router), but it performed far worse than
| Qwen2.5-Coder-7B (but sure, 1.5B vs 7B)
| sparacha wrote:
| Can you share more about your evaluation setup? I would love to
| see the specific usage pattern as we have tested our model
| against smaller LLMs and foundational models and our results
| show things differently. Of course, routing policies should
| follow best practices here:
| https://docs.archgw.com/guides/llm_router.html
|
| Nonetheless, super curious to learn more and see what we may be
| able to improve. This is technically not a classifier model -
| its a usage prediction model (feels like a classifier, but not
| quite in terms of intended usage)
| cotran2 wrote:
| According to the post, the model is fine-tuned for routing to
| different tasks/domains. Classifying difficulty level is
| probably not the intended use case.
| jgant13 wrote:
| Solid. Can you show us when to use this vs. say OpenRouter? The
| performance seems strong for sure. TIA.
| sparacha wrote:
| Arch is developer friendly, but designed for enterprise-grade
| customers in mind. The core contributors of Envoy redesigned
| the proxy substrate to handle prompts - offering something that
| is battle tested in terms of resiliency, speed, and
| deployments. Second, OpenRouter offers choice of models, but
| dynamically routing to LLMs based on user-defined usage
| policies is uniquely available in Arch. Hope that helps
___________________________________________________________________
(page generated 2025-07-01 23:01 UTC)