[HN Gopher] The von Neumann bottleneck is impeding AI computing?
___________________________________________________________________
The von Neumann bottleneck is impeding AI computing?
Author : Nezteb
Score : 25 points
Date : 2025-09-26 21:12 UTC (1 hours ago)
(HTM) web link (research.ibm.com)
(TXT) w3m dump (research.ibm.com)
| mwkaufma wrote:
| The old saw from corporations that want to sell you an locked-
| down alternative to general-purpose computing -- now for "AI"
| bahmboo wrote:
| Huh, I did not get that from the article. The main takeaway for
| me was doing ALU operations in memory resulting in massive
| energy savings. There is still a von Neumann architecture
| running the show.
| nyrikki wrote:
| Nit,
|
| ARM processors primarily use a modified Harvard architecture,
| including the raspberry pi pico.
| NooneAtAll3 wrote:
| this isn't about Harvard/VonNeuman split/no-split between
| i-cache and d-cache
|
| I think this post is more about... compute in memory? if I got
| it right?
| danudey wrote:
| Sort of? It's about locality of data; this has often been a
| bottleneck, which is why we have CPU caches to keep data
| extremely close to the CPU cores with practically zero
| latency and throughput limitations compared to fetching from
| main memory. Unfortunately now we're shuffling terabytes of
| data through our algorithms and the CPU spends a huge amount
| of its time waiting for the next batch of data to come in
| through the pipe.
|
| This is, IIRC, part of why Apple's M-series chips are as
| performant as they are: they not only have a unified memory
| architecture which eliminates the need to copy data from CPU
| main memory to GPU or NPU main memory to operate on it (and
| then copy the result back) but the RAM being on the package
| means that it's slightly "more local" and the memory channels
| can be optimized for the system they're going to be connected
| to.
| bobmcnamara wrote:
| Nit: RP2040 is a Von Neumann. There's only one AHB port on the
| m0.
|
| Edit: see also ARM7TDMI, Cortex-m0/0+/1, and probably a few
| others. All the big stuff is modified Harvard or very rarely
| pure Harvard.
| Animats wrote:
| Actual result: "This new process promises to increase the number
| of optical fibers that can be connected at the edge of a chip, a
| measure known as beachfront density, by six times."
|
| Faster interconnects are always nice, but this is more like
| routine improvement.
| bahmboo wrote:
| "In recent inference tests run on a 3-billion-parameter LLM
| developed from IBM's Granite-8B-Code-Base model, NorthPole was
| 47 times faster than the next most energy-efficient GPU and was
| 73 times more energy efficient than the next lowest latency
| GPU."
|
| It's also fascinating that they are experimenting with analog
| memory because it pairs so well with model weights
| lomase wrote:
| Why they don't use AI to create a new architecture?
| observationist wrote:
| https://github.com/GAIR-NLP/ASI-Arch
|
| This is being done, with great results so far. As models get
| better, architecture search and creation and refinment
| improves, driving a reinforcement loop. At some point in the
| near future the big labs will likely start seeing significant
| returns from methods like this, translating into better and
| faster AI for consumers.
| lomase wrote:
| I skimed the repo and only found sloop. Can you point out
| where I can find those new architectures you talk about?
| jedberg wrote:
| Do you want SkyNet? That's how you get SkyNet.
| amelius wrote:
| No, that's how you get SlopNet.
| owyn wrote:
| If you follow the press release rabbit a few clicks, there's an
| article in Science describing the architecture in more detail:
|
| https://www.science.org/doi/full/10.1126/science.adh1174
|
| Also they've been working on this for 10+ years so it's not
| exactly new news.
| stego-tech wrote:
| IBM initially leads with the more salient point (current
| architecture designs are hindering frontier computing concepts),
| then just kinda...relents into iterative improvement.
|
| Which is fine! I am all for iterative improvements, it's how we
| got to where we are today. I just wish more folks would start
| openly admitting that our current architecture designs are
| broadly based off "low hanging fruit" of early electronics and
| microprocessors, followed by a century of iterative improvements.
| With the easy improvements already done and universally
| integrated, we're stuck at a crossroads:
|
| * Improve our existing technologies iteratively and hope we break
| through some barrier to achieve rapid scaling again
|
| OR
|
| * Accept that we cannot achieve new civilizational uplifts with
| existing technologies, and invest more capital into frontier R&D
| (quantum processing, new compute substrates, etc)
|
| I feel like our current addiction to the AI CAPEX bubble is a
| desperate Hail Mary to validate our current tech as the only way
| forward, when in fact we haven't really sufficiently explored
| alternatives in the modern era. I could very well be wrong, but
| that's the read I get from the hardware side of things and
| watching us backslide into the 90s era of custom chips to achieve
| basic efficiency gains again.
___________________________________________________________________
(page generated 2025-09-26 23:00 UTC)