[HN Gopher] Launch HN: IonRouter (YC W26) - High-throughput, low...
___________________________________________________________________
Launch HN: IonRouter (YC W26) - High-throughput, low-cost inference
Hey HN -- I'm Veer and my cofounder is Suryaa. We're building
Cumulus Labs (YC W26), and we're releasing our latest product
IonRouter (https://ionrouter.io/), an inference API for open-source
and fine tuned models. You swap in our base URL, keep your existing
OpenAI client code, and get access to any model (open source or
finetuned to you) running on our own inference engine. The problem
we kept running into: every inference provider is either fast-but-
expensive (Together, Fireworks -- you pay for always-on GPUs) or
cheap-but-DIY (Modal, RunPod -- you configure vLLM yourself and
deal with slow cold starts). Neither felt right for teams that just
want to ship. Suryaa spent years building GPU orchestration
infrastructure at TensorDock and production systems at Palantir. I
led ML infrastructure and Linux kernel development for Space Force
and NASA contracts where the stack had to actually work under
pressure. When we started building AI products ourselves, we kept
hitting the same wall: GPU infrastructure was either too expensive
or too much work. So we built IonAttention -- a C++ inference
runtime designed specifically around the GH200's memory
architecture. Most inference stacks treat GH200 as a compatibility
target (make sure vLLM runs, use CPU memory as overflow). We took a
different approach and built around what makes the hardware
actually interesting: a 900 GB/s coherent CPU-GPU link, 452GB of
LPDDR5X sitting right next to the accelerator, and 72 ARM cores you
can actually use. Three things came out of that that we think are
novel: (1) using hardware cache coherence to make CUDA graphs
behave as if they have dynamic parameters at zero per-step cost --
something that only works on GH200-class hardware; (2) eager KV
block writeback driven by immutability rather than memory pressure,
which drops eviction stalls from 10ms+ to under 0.25ms; (3)
phantom-tile attention scheduling at small batch sizes that cuts
attention time by over 60% in the worst-affected regimes. We wrote
up the details at cumulus.blog/ionattention. On multimodal
pipelines we get better performance than big players (588 tok/s vs.
Together AI's 298 on the same VLM workload). We're honest that p50
latency is currently worse (~1.46s vs. 0.74s) -- that's the
tradeoff we're actively working on. Pricing is per token, no idle
costs: GPT-OSS-120B is $0.02 in / $0.095 out, Qwen3.5-122B is $0.20
in / $1.60 out. Full model list and pricing at
https://ionrouter.io. You can try the playground at
https://ionrouter.io/playground right now, no signup required, or
drop your API key in and swap the base URL -- it's one line. We
built this so teams can see the power of our engine and eventually
come to us for their finetuned model needs using the same solution.
We're curious what you think, especially if you're running
finetuned or custom models -- that's the use case we've invested
the most in. What's broken, what would make this actually useful
for you?
Author : vshah1016
Score : 35 points
Date : 2026-03-12 18:52 UTC (4 hours ago)
(HTM) web link (ionrouter.io)
(TXT) w3m dump (ionrouter.io)
| GodelNumbering wrote:
| As an inference hungry human, I am obviously hooked. Quick
| feedback:
|
| 1. The models/pricing page should be linked from the top perhaps
| as that is the most interesting part to most users. You have
| mentioned some impressive numbers (e.g. GLM5~220 tok/s $1.20 in *
| $3.50 out) but those are way down in the page and many would miss
| it
|
| 2. When looking for inference, I always look at 3 things: which
| models are supported, at which quantization and what is the
| cached input pricing (this is way more important than headline
| pricing for agentic loops). You have the info about the first on
| the site but not 2 and 3. Would definitely like to know!
| 2uryaa wrote:
| Thank you for the feedback! I think we will definitely redo the
| info on the frontpage to reorg and show quantizations better.
| For reference, Kimi and Minimax are NVFP4. The rest are FP8.
| But I will make this more obvious on the site itself.
| Oras wrote:
| The problem is well articulated and nice story for both
| cofounders.
|
| One thing I don't get is why would anyone use a direct service
| that does the same thing as others when there are services such
| as openrouter where you can use the same model from different
| providers? I would understand if your landing page mentioned
| fine-tuning only and custom models, but just listing same open
| source models, tps and pricing wouldn't tell me how you're
| different from other providers.
|
| I remember using banana.dev a few years ago and it was very clear
| proposition that time (serverless GPU with fast cold start)
|
| I suppose positioning will take multiple iterations before you
| land on the right one. Good luck!
| 2uryaa wrote:
| Hey Oras, thank you for the feedback! I think we definitely
| could list on OpenRouter but as you point out, our end goal is
| to host finetuned models for individuals. The IonRouter product
| is mostly to showcase our engine. In the backend, we are
| multiplexing finetuned and open source models on a homogenous
| fleet of GPUs. So if you feel better or even similar
| performance difference on our cloud, we're already proving what
| we set out to show.
|
| I do think we will lean harder into the hosting of fine-tuned
| models though, this is a good insight.
| reactordev wrote:
| "Pricing is per token, no idle costs: GPT-OSS-120B is $0.02 in /
| $0.095 out, Qwen3.5-122B is $0.20 in / $1.60 out. Full model list
| and pricing at https://ionrouter.io."
|
| Man you had me panicking there for a second. Per token?!? Turns
| out, it's per million according to their site.
|
| Cool concept. I used to run a Fortune 500's cloud and GPU
| instances hot and ready were the biggest ask. We weren't ready
| for that, cost wise, so we would only spin them up when
| absolutely necessary.
| 2uryaa wrote:
| Haha sorry for the typo! Your F500 use case is exactly who we
| want to target, especially as they start serving finetunes on
| their own data. Thanks for the feedback!
| nylonstrung wrote:
| Unless I misunderstood it seems like this is trailing the pareto
| frontier in cost and speed.
|
| Compare to providers like Fireworks and even with the openrouter
| 5% charge it's not competitive
| 2uryaa wrote:
| our SLA is actually higher and we are lower priced. We are also
| using this as a step into serving finetuned models for much
| cheaper than Fireworks/Together and not having the horrible
| cold starts of Modal. We're essentially trying to prove that
| our engine can hang with the best providers while multiplexing
| models.
| erichocean wrote:
| > _what would make this actually useful for you?_
|
| A privacy policy that's at least as good as Vertex.ai at Google.
|
| Otherwise it's a non-starter at any price.
| Oras wrote:
| What's unique about Vertex's privacy policy?
| 2uryaa wrote:
| Also curious about this. We have a 30 day content retention
| policy and have to have access to your fine-tuned model/LoRa if
| deploying that. If there's anything we can change, happy to
| hear it out.
| cmrdporcupine wrote:
| Very cool, I see that "Deploy your finetunes, custom LoRAs, or
| any open-source model on our fleet." is _" Book a call"_ -- any
| sense of what pricing will actually look like here, since this
| seems like it's kind of where your approach wins out, the ability
| to swap in custom model easier/cheaper?
|
| Just curious how close we are to a world where I can fine tune
| for my (low volume calls) domain and then get it hosted. Right
| now this is not practical anywhere I've seen, at the volumes I
| would be doing it at (which are really hobby level).
| 2uryaa wrote:
| We usually charge by GPU hour for those finetunes, around 8-10
| dollars depending on GPU type and volume! This is similar to
| Modal, but since the engine is fully ours, you don't wait ~1
| min for cold starts. Ideally, we will make onboarding super
| frictionless and self serve, but onboarding people manually for
| now.
| Frannky wrote:
| I have no idea how much the demand is for fine-tuned models. Is
| it big? Are people actively looking for endpoints for fine-tuned
| models? Why? Mostly out of curiosity, I personally never had the
| need.
|
| What I want from an LLM is smart, super cheap, fast, and private.
| I wonder if we will ever get there. Like having a cheap Cerebras
| machine at home with oss 400B models on it.
___________________________________________________________________
(page generated 2026-03-12 23:00 UTC)