[HN Gopher] Challenges and Research Directions for Large Languag...
___________________________________________________________________
Challenges and Research Directions for Large Language Model
Inference Hardware
Author : transpute
Score : 108 points
Date : 2026-01-25 02:48 UTC (20 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| jauntywundrkind wrote:
| > _To address these challenges, we highlight four architecture
| research opportunities:_ High Bandwidth Flash _for 10X memory
| capacity with HBM-like bandwidth;_ Processing-Near-Memory _and_
| 3D memory-logic stacking _for high memory bandwidth; and_ low-
| latency interconnect _to speedup communication._
|
| High Bandwidth Flash (HBF) got submitted 6 hours ago! It's a
| _great_ article, fantastic coverage of a wide section of the
| rapidly moving industry.
| https://news.ycombinator.com/item?id=46700384
| https://blocksandfiles.com/2026/01/19/a-window-into-hbf-prog...
|
| HBF is about having many dozens or hundreds of channels of flash
| memory. The idea of having Processing Near HBF, spread out,
| perhaps in mixed 3d design, would be not at all surprising to me.
| One of the main challenges for HBF is building improved vias,
| improved stacking, and if that tech advanced the idea of more
| mixed NAND and compute layers rather than just NAND stacks
| perhaps opens up too.
|
| This is all really exciting possible next steps.
| amelius wrote:
| Why is persistence such a big thing here? Non-flash memory just
| needs a tiny bit of power to keep its data. I don't see the
| revolutionary usecase.
| Gracana wrote:
| Density is the key here, not persistence.
| amelius wrote:
| Thanks! This explains it.
|
| Now I'm wondering how you deal with the limited number of
| write cycles of Flash memory. Or maybe that is not an issue
| in some applications?
| mrob wrote:
| During inference, most of the memory is read only.
| amelius wrote:
| Sounds fair. That's not the kind of machine I'd want as a
| development system though. And usually development
| systems are beefier than production systems. So curious
| how they'd solve that.
| Gracana wrote:
| Yeah, it is quite specialized for inference. It's
| unlikely that you'd see this stuff outside of hardware
| specifically for that.
|
| Development systems for AI inference tend to be smaller
| by necessity. A DGX Spark, Station, a single B300 node...
| you'd work on something like that before deploying to a
| larger cluster. There's just nothing bigger than what
| you'd actually deploy to.
| transpute wrote:
| HBF, like expensive HBM, is targeted at AI data centers.
| The KAIST professor discussed an HBF unit having a
| capacity of 512 GB and a 1.638 TBps bandwidth.
|
| PCIe x8 GPU bandwidth is about 32GBbps, so HBF could be
| 50x PCIe bandwidth.
| bluehat974 wrote:
| Related too https://www.sdxcentral.com/news/ai-inference-crisis-
| google-e...
| random_duck wrote:
| Yup, reads like the executive summary (in a good way).
| random3 wrote:
| David Patterson is such a legend! From RAID to RISC and one of
| the best books in computer architecture, he's on my personal hall
| of fame.
|
| Several years ago I was at one of the Berkley AMP Lab retreats at
| Asilomar, and as I was hanging out, I couldn't figure how I know
| this person in front of me, until an hour later when I saw his
| name during a panel :)).
|
| It was always the network. And David Patterson, after RISC,
| started working on iRAM, that was tackling a related problem.
|
| NVIDIA bought Mellanox/Infiniband, but Google has historically
| excelled at networking, and the TPU seems to be designed to scale
| out in the best possible way.
| suggeststrongid wrote:
| Can't we credit the first author in the title too? Come on.
| random_duck wrote:
| No we can't, that would be a crime against royalty :)
| transpute wrote:
| The current title uses 79 characters of 80 character budget:
| 75% = title written by first author 22% = name of second
| author, endorsing work of first author
|
| HN mods can revert the title to the original headline, without
| any author.
| amelius wrote:
| That appendix of memory prices looks interesting, but misses the
| recent trend.
| zozbot234 wrote:
| Weird to see no mention in this paper of persistent memory
| technologies beyond NAND flash. Some of them, like ReRAM, also
| enable compute-in-memory which the authors regard as quite
| important.
| HPsquared wrote:
| Why not, instead of passing the entire model through a processor
| and running it on every bit of data, pass the data (which is much
| smaller) through the model? As in, have compute and memory
| together in the silicon. Then you only need to shuffle the data
| itself around (perhaps by broadcast) rather than the entire
| model. That seems like it would use a LOT less energy.
|
| Or is it not possible to make the algorithms parallel to this
| degree?
|
| Edit: apparently this is called "compute-in-memory"
| pavpanchekha wrote:
| Frontier models are now much bigger than an individual query,
| hence batching, MoE, etc. So this idea, while very plausible,
| has economic constraints, you'd need vast amounts of memory.
| jmalicki wrote:
| This is done that way at the GPU layer of abstraction -
| generally (with some exceptions!) the model lives in GPU vram,
| and you stream the data batch by batch through the model.
|
| The problem is that for larger models the model barely fits in
| VRAM, so it definitely doesn't fit in cache.
|
| Dataflow processors like cerebras do stream the data through
| the model (for smaller models at least, or if they can have
| smaller portions of models) - each little core has local memory
| and you move the data to where it needs to go. To achieve this
| though, Cerebras has 96GB of what is basically L1 cache among
| its cores, which is... a lot of SRAM.
| westurner wrote:
| In-memory processing: https://en.wikipedia.org/wiki/In-
| memory_processing
|
| Computational RAM:
| https://en.wikipedia.org/wiki/Computational_RAM
| westurner wrote:
| Designing a concept sustainable RAM product and in working
| around multiplexing scaling challenges I somewhat accidentally
| developed a potential solution for hosting already-trained LLMs
| with very low energy and hardware in carbon and lignin;
|
| > _You have effectively designed a Diffractive Deep Neural
| Network (D^2NN) that doubles as a storage device._
|
| Mode Division Multiplexing (MDM) via OAM Solitons potentially
| with gratings designed with Inverse Design of a Transition Map
| to be lasered possibly with a Galvo Laser. This would be a very
| low power way to run LLMs; on a lasered substrate
___________________________________________________________________
(page generated 2026-01-25 23:01 UTC)