[HN Gopher] Hypura - A storage-tier-aware LLM inference schedule...
___________________________________________________________________
Hypura - A storage-tier-aware LLM inference scheduler for Apple
Silicon
Author : tatef
Score : 177 points
Date : 2026-03-24 16:02 UTC (6 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| marksully wrote:
| Where does "1T parameter model" come from? I can only see models
| with 70B params or less mentioned in the repo.
| causal wrote:
| Yeah title comes from nowhere in the link. No doubt it's
| possible but all that matters is speed and we learn nothing of
| that here...
| tatef wrote:
| I'm referencing it as being possible, however I didn't share
| benchmarks because candidly the performance would be so slow it
| would only be useful for very specific tasks over long time
| horizons. The more practical use cases are less flashy but
| capable of achieving multiple tokens/sec (ie smaller MoE models
| where not all experts need to be loaded in memory
| simultaneously)
| zozbot234 wrote:
| It will be interesting to compare this to
| https://news.ycombinator.com/item?id=47476422 and
| https://news.ycombinator.com/item?id=47490070 . Very similar
| design except that this is apparently using mmap, which according
| to the earlier experiment incurs significant overhead.
| salynchnew wrote:
| It was written by an LLM, so... yeah.
| jeffybefffy519 wrote:
| Except this isnt using heavily quantised versions of the model
| thus reducing quality.
| Insanity wrote:
| This is a pretty cool project! Essentially this is like using
| Swap memory to extend your RAM, but in a 'smart' way so you don't
| overload the NVMe unnecessarily.
|
| I do wonder in practice how the 'smarts' pan out, because putting
| a ton of stress on your NVMe during generation is probably not
| the best choice for it's longevity.
| zozbot234 wrote:
| This is not putting any stress or wear on the NVMe, it's a pure
| read workload.
| tatef wrote:
| Yes, exactly this.
| embedding-shape wrote:
| > but in a 'smart' way so you don't overload the NVMe
| unnecessarily
|
| "overloading NVMe"? What is that about? First time I've heard
| anything about it.
|
| > because putting a ton of stress on your NVMe during
| generation
|
| Really shouldn't "stress your NVMe", something is severely
| wrong if that's happening. I've been hammering my SSDs forever,
| and while write operations "hurt" the longevity of the flash
| cells themselves, the controller interface really shouldn't be
| affected by this at all, unless I'm missing something here.
| Insanity wrote:
| I had assumed heat generation on the controller if it's
| continuously reading. But maybe it's not actually bad.
| throwway120385 wrote:
| Just pop a heatsink on it and call it good.
| tatef wrote:
| Hypura reads tensor weights from the GGUF file on NVMe into
| RAM/GPU memory pools, then compute happens entirely in
| RAM/GPU.
|
| There is no writing to SSDs on inference with this
| architecture.
| embedding-shape wrote:
| Even if there was a ton of writing, I'm not sure where NVMe
| even comes in the picture, write durability is about the
| flash cells on SSDs, nothing to do with the interface,
| someone correct me if I'm wrong.
| hrmtst93837 wrote:
| People talk about "SSD endurance", but enough parallel I/O on
| M1/M2 can make the NVMe controller choke, with very weird
| latncy spikes.
| monksy wrote:
| There needs to be something like this from Ollama. At the moment
| Ollama has a lot of flaws that prevent it from getting great
| performance. (My understanding is better GPU/CPU splits, etc).
| But Ollama is the only way to host an LLM and have it switch out
| on demand. Sigh.
| rubiquity wrote:
| llama.cpp and llama-swap do this better than Ollama and with
| far more control.
| circularfoyers wrote:
| Don't even need to use llama-swap anymore now that llama-
| server supports the same functionality.
| zozbot234 wrote:
| Ollama has very substandard support for mmap at present, which
| hurts inference with larger models. There are some recent pull
| requests in flight that should help address this to at least
| some extent https://github.com/ollama/ollama/pull/14525
| https://github.com/ollama/ollama/pull/14134
| https://github.com/ollama/ollama/pull/14864 but progress seems
| to be stalling. Their support for recent Qwen models seems to
| also have some bespoke incompatibilities with llama.cpp, which
| doesn't help matters; it's difficult to test the same model
| with both.
| baq wrote:
| Intel Optane rolling in its grave.
| liuliu wrote:
| Still have 4 brand new ones in my storage unit. Just in case
| these moments.
|
| Joke aside (I do have them tho!), I don't think Optane is that
| much use (not to mention it is only 256GiB for my unit). It is
| useful legacy crutch if you have legacy software that is not
| designed to issue multiple reads / writes in parallel. If you
| do, it is really not faster than NVMe, especially these modern
| ones.
| zozbot234 wrote:
| It's not about being faster (except for small reads where
| latency dominates, which is actually relevant when reading a
| handful of expert-layers immediately after routing), it's the
| wearout resistance which opens up the possibility of storing
| KV-cache (including the "linear" KV-cache of recent Qwen,
| which is not append-only as it was with the pure attention
| model) and maybe even per-layer activations - though this has
| the least use given how ephemeral these are.
| speedgoose wrote:
| Is it too late for Intel to bring them back to life?
| c0balt wrote:
| Yes, their NAND division has been sold, it is now mostly
| under solidigm. Maybe solidigm could bring it back, but it
| seems unlikely (given the previous commercial failure).
| walterbell wrote:
| Nvidia and SK Hynix are bringing HBF to market for $$.
| 0ptan3 wrote:
| pmem
| moffkalast wrote:
| Wouldn't be Intel if they didn't quit halfway through on a good
| thing.
|
| Still, couldn't one get a RAID 0 card with four drives to
| saturate a 16x lane? That's already the max one could push
| through PCIe anyhow.
| aitchnyu wrote:
| Memristors are also missing in this AI hype even when they were
| around the corner 10 years back.
| nullbyte wrote:
| I am curious how the TPS compares vs default OS virtual memory
| paging
| anshulbasia27 wrote:
| OS paging would be significantly worse here. The kernel's page
| fault handler is reactive -- it doesn't know you're about to read
| layer 47's FFN weights, so it can't prefetch. You stall on every
| fault, wait for the 4KB/16KB page to load, then resume. With 80
| layers of dense FFN streaming, that's thousands of cold faults
| per token. What makes this approach faster is
| that the model's access pattern is completely deterministic
| during inference. You know exactly which tensors
| are needed next because transformer layers execute sequentially.
| So you can issue large sequential reads and prefetch the
| next layer while the current one is computing on Metal.
| The OS page cache can't do that -- it has no concept of "layer
| N+1 comes after layer N." For MoE it's even more
| stark. The OS would page in all 8 experts on the first token that
| routes to each one, then evict them under memory pressure
| with LRU, which has no idea that expert 3 fires 10x more often
| than expert 7. The neuron cache here is basically a domain-
| specific replacement policy.
| zozbot234 wrote:
| > The kernel's page fault handler is reactive -- it doesn't
| know you're about to read layer 47's FFN weights, so it can't
| prefetch.
|
| man 2 madvise
| astrange wrote:
| That works for readahead but it's not good for random access.
| readv, aio, dispatch_io are better there.
| zozbot234 wrote:
| This claim is a bit apples and oranges (no pun intended!).
| madvise is all about providing hints to the kernel to tune
| the page cache and readahead (including possibly disabling
| readahead altogether). it's not about performing reads into
| private memory buffers, which is actually where the options
| you mentioned fit in.
| EnPissant wrote:
| That assumes you have significant work to do between fetches
| (so you can prefetch while using the current data). With LLM
| decode you don't.
| EnPissant wrote:
| You do not provide any comparison to llama.cpp with mmap.
|
| You do not explain how any kind of predictor can work for MoE
| experts.
|
| You do not explain how prediction can even be useful. I can
| predict the layers used in a dense model (all of them are used in
| order), but that doesn't help me much. It's still bottlenecked on
| bandwidth (hint: MoE doesn't change this).
| amelius wrote:
| This is <1 tok/s for the 40GB model.
|
| Come on, "Run" is not the right word. "Crawl" is.
|
| Headlines like that are misleading.
| smlacy wrote:
| Yes, and with virtually zero context, which makes an enormous
| difference for TTFT on the MoE models.
| feznyng wrote:
| Could still be useful; maybe for overnight async workloads?
| Tell your agent research xyz at night and wake up to a report.
| maleldil wrote:
| Assuming 1 token per second and "overnight" being 12 hours,
| that's 43 200 tokens. I'm not sure what you can meaningfully
| achieve with that.
| zozbot234 wrote:
| Sure, but if long-term throughput is a real limitation
| there's plenty of ways to address that while still not
| needing to keep anywhere close to _all_ model weights in
| RAM (which is still the conventional approach with MoE). So
| the gain of a smaller memory footprint is quite real.
| speedgoose wrote:
| I wonder how many minutes per token on GLM 5.
| vicchenai wrote:
| the practical question is whether the read pattern is sequential
| enough to actually saturate nvme bandwidth or if the attention
| layer access pattern ends up being random enough to kill
| throughput. sequential reads on a decent nvme get you 5-7 GB/s,
| random reads drop to maybe 500 MB/s depending on queue depth.
|
| for a 1T model youd need to stream something like 2TB of weights
| per forward pass at fp16. even at peak sequential thats 300+
| seconds per token which is... not great for interactive use but
| maybe fine for batch inference where you dont care about latency.
|
| still a cool proof of concept though. the gap between 'can run'
| and 'runs usefully' is where things get interesting.
| zozbot234 wrote:
| > for a 1T model youd need to stream something like 2TB of
| weights per forward pass
|
| Isn't this missing the point of MoE models completely? MoE
| inference is sparse, you only read a small fraction of the
| weights per layer. You still have a problem of each individual
| expert-layer being quite small (a few MiBs each give or take)
| but those reads are large enough for the NVMe.
| visarga wrote:
| But across a sequence you still have to load most of them.
| tatef wrote:
| Yes, definitely agree. It's more of a POC than a functional use
| case. However, for many smaller MoE models this method can
| actually be useful and capable of achieving multiple
| tokens/sec.
| p_ing wrote:
| 4K random read with a queue depth of 1 on an M1 Max is about
| 65MB/s.
| erikcw wrote:
| Simon Willison wrote a good post about Dan Woods' work on
| "Autoresearching Apple's "LLM in a Flash" to run Qwen 397B
| locally".
|
| [0] https://simonwillison.net/2026/Mar/18/llm-in-a-flash/
| root_axis wrote:
| Are there any 1T parameter open source models?
| zozbot234 wrote:
| Kimi 2.5?
| root_axis wrote:
| Thanks, TIL.
| ai-inquisitor wrote:
| That model is "open weight", not open source. We have no idea
| what data Moonshot trained on.
| vanyaland wrote:
| For a lot of local workloads, sub-1 tok/s is useless in
| foreground and perfectly acceptable in background. If the choice
| is "this crashes" vs "this finishes overnight," that's still a
| meaningful capability jump.
| shubhamintech wrote:
| The MoE point matters here ie sparse activation means you're not
| reading all 2TB per forward pass, but the access pattern flips
| from sequential to random which is exactly the worst case for
| NVMe. Been thinking about this a lot for agent inference
| workloads where you want consistent latency more than peak
| throughput.
| simonw wrote:
| Suggestion for the maintainers: the comparison table currently
| lists some pretty old models, Qwen 2.5 14B and Mixtral 8x7B and
| Llama 3.3 70B.
|
| A lot of people are reporting incredible results with the Qwen
| 3.5 MoE models on Apple hardware right now (streaming experts -
| see https://simonwillison.net/2026/Mar/24/streaming-experts/) -
| it would be great to get some of those models into that table.
|
| Maybe the 1T parameter Kimi K2.5 too if you can get that to work,
| see https://twitter.com/seikixtc/status/2036246162936910322 and
| https://twitter.com/danpacary/status/2036480556045836603
| Imustaskforhelp wrote:
| Simon, A little offtopic but it seems that your website isn't
| working.
|
| > An error occurred in the application and your page could not
| be served. If you are the application owner, check your logs
| for details. You can do this from the Heroku CLI with the
| command
|
| I get this error when I go to simonwillison.net
|
| Any random blog/link works for example though:
| https://simonwillison.net/2026/Mar/19/openai-acquiring-astra...
|
| (I checked your website because I wanted to see if you had
| written something about trivy/litellm as well, I highly
| recommend checking out what has happened within litellm space
| if possible as I would love to read your thoughts on it)
|
| Have a nice day simon!
|
| Edit: now the website works but I am not sure what had gone
| wrong previously, (an issue from heroku maybe?) as its working
| now
|
| Edit-2: after the website working, I am able to see that you
| have already made a post about it.
| abtinf wrote:
| The lack of a token rate metric for the kimi example is
| disappointing.
| zozbot234 wrote:
| The latter link says they get ~1.7 tok/s which is quite
| impressive for a near-SOTA local model running on ordinary
| hardware.
| tatef wrote:
| Thanks for sharing this! If you'd be interested in running the
| benchmark yourself with Hypura I'd happily merge into our
| stats. Otherwise will add to my todo list :)
| astrange wrote:
| > Consumer hardware (MacBook Pro, Mac Studio) ships with fast
| unified memory and NVMe storage, but limited capacity. A 32 GB M1
| Max cannot naively load a 40 GB model -- the OS will swap-thrash
| until the OOM killer intervenes.
|
| macOS doesn't have an "OOM killer" in that sense. (It has an out
| of swap space killer but it's pretty weak.)
|
| So what will happen is, either your memory wiring will fail, or
| else it will get really slow and panic.
___________________________________________________________________
(page generated 2026-03-24 23:00 UTC)