[HN Gopher] I want a good parallel computer
___________________________________________________________________
I want a good parallel computer
Author : raphlinus
Score : 51 points
Date : 2025-03-21 19:55 UTC (3 hours ago)
(HTM) web link (raphlinus.github.io)
(TXT) w3m dump (raphlinus.github.io)
| armchairhacker wrote:
| > The GPU in your computer is about 10 to 100 times more powerful
| than the CPU, depending on workload. For real-time graphics
| rendering and machine learning, you are enjoying that power, and
| doing those workloads on a CPU is not viable. Why aren't we
| exploiting that power for other workloads? What prevents a GPU
| from being a more general purpose computer?
|
| What other workloads would benefit from a GPU?
|
| Computers are so fast that in practice, many tasks don't need
| more performance. If a program that runs those tasks is slow,
| it's because that program's code is particularly bad, and the
| solution to make the code less bad is simpler than re-writing it
| for the GPU.
|
| For example, GUIs have been imperceptibly reactive to user input
| for over 20 years. If an app's GUI feels sluggish, the problem is
| that the app's actions and rendering aren't on separate
| coroutines, or the action's coroutine is blocking (maybe it needs
| to be on a separate thread). But the rendering part of the GUI
| doesn't need to be on a GPU (any more than it is today, I admit I
| don't know much about rendering), because responsive GUIs exist
| today, some even written in scripting languages.
|
| In some cases, parallelizing a task intrinsically makes it
| slower, because the number of sequential operations required to
| handle coordination mean there are more forced-sequential
| operations in total. In other cases, a program spawns 1000+
| threads but they only run on 8-16 processors, so the program
| would be faster if it spawned less threads because it would still
| use all processors.
|
| I do think GPU programming should be made much simpler, so this
| work is probably useful, but mainly to ease the implementation of
| tasks that already use the GPU: real-time graphics and machine
| learning.
| wmf wrote:
| A big one is video encoding. It seems like GPUs would be ideal
| for it but in practice limitations in either the hardware or
| programming model make it hard to efficiently run on GPU shader
| cores. (GPUs usually include separate fixed-function video
| engines but these aren't programmable to support future
| codecs.)
| dist-epoch wrote:
| Video encoding is done with fixed-function for power
| efficiency. A new popular codec like H26x codec appears every
| 5-10 years, there is no real need to support future ones.
| IshKebab wrote:
| Having worked for a company that made a "hundreds of small CPUs
| on a single chip", I can tell you now that they're all going to
| fail because the programming model is too weird, and nobody will
| write software for them.
|
| Whatever comes next will be a GPU with extra capabilities, not a
| totally new architecture. Probably an nVidia GPU.
| bryanlarsen wrote:
| While acknowledging that it's theoretically possible other
| approaches might succeed, it seems quite clear the author
| agrees with you.
| convolvatron wrote:
| my take from reading this is more about programming
| abstractions than any particular hardware instantiation. the
| part of the Connection Machine that remains interesting is not
| building machines with CPUS with transistor counts in the
| hundreds running off a globally synchronous clock, but that
| there were a whole family of SIMD languages and let you do
| general purpose programming in parallel. And that those
| language were still relevant when the architecture changed to a
| MIMD machine with a bunch of vector units behind each CPU.
| snovymgodym wrote:
| Reminds me of Itanium
| turtletontine wrote:
| Could you elaborate on this? How does many-small-CPUs make for
| a weirder programming model than a GPU?
|
| Im no expert, but I've done my fair share of parallel HPC stuff
| using MPI, and a little bit of Cuda. And to me the GPU
| programming model is far far "weirder" and harder to code for
| than the many-CPUs model. (Granted, I'm assuming you're
| describing a different regime?)
| dist-epoch wrote:
| In CUDA you don't really manage the individual compute units,
| you start a kernel, and the drivers take care of distributing
| that to the compute cores and managing the data flows between
| them.
|
| When programming CPUs however you are controlling and
| managing the individual threads. Of course, there are
| libraries which can do that for you, but fundamentally it's a
| different model.
| mikewarot wrote:
| The key transformation required to make any parallel
| architecture work is going to be taking a program that humans
| can understand, and translating it into a _directed acyclic
| graph_ of logical Boolean operations. This type of intermediate
| representation could then be broken up into little chunks for
| all those small CPUS. It could be executed _very slowly_ using
| just a few logic gates and enough ram to hold the state, or it
| could run at FPGA speeds or better on a generic sea of LUTs.
| KerrAvon wrote:
| Isn't that the Connection Machine architecture?
| svmhdvn wrote:
| I've always admired the work that the team behind
| https://www.greenarraychips.com/ does, and the GA144 chip seems
| like a great parallel computing innovation.
| bee_rider wrote:
| It is odd that he talks about Larabee so much, but doesn't
| mention the Xeon Phis. (Or is it Xeons Phi?).
|
| > As a general trend, CPU designs are diverging into those
| optimizing single-core performance (performance cores) and those
| optimizing power efficiency (efficiency cores), with cores of
| both types commonly present on the same chip. As E-cores become
| more prevalent, algorithms designed to exploit parallelism at
| scale may start winning, incentivizing provision of even larger
| numbers of increasingly efficient cores, even if underpowered for
| single-threaded tasks.
|
| I've always been slightly annoyed by the concept of E cores,
| because they are so close to what I want, but not quite there...
| I want, like, throughput cores. Let's take E cores, give them
| their AVX-512 back, and give them higher throughput memory. Maybe
| try and pull the Phi trick of less OoO capabilities but more
| threads per core. Eventually the goal should be to come up with
| an AVX unit so big it kills iGPUs, haha.
| nullpoint420 wrote:
| I've always wondered if you could use iGPU compute cores with
| unified memory as "transparent" E-cores when needed.
|
| Something like OpenCL/CUDA except it works with
| pthreads/goroutines and other (OS) kernel threading primitives,
| so code doesn't need to be recompiled for it. Ideally the OS
| scheduler would know how to split the work, similar to how
| E-core and P-core scheduling works today.
|
| I don't do HPC professionally, so I assume I'm ignorant to why
| this isn't possible.
| Retr0id wrote:
| Isn't Xeon Phi just an instance of Larrabee?
| andrewstuart wrote:
| AMD Strix Halo APU is a CPU with very powerful integrated GPU.
|
| It's faster at AI than an Nvidia RTX4090, because 96GB of the
| 128GB can be allocated to the GPU memory space. This means it's
| doesn't have the same swapping/memory thrashing that a discrete
| GPU experiences when processing large models.
|
| 16 CPU cores and 40 GPU compute units sounds pretty parallel to
| me.
|
| Doesn't that fit the bill?
| bigyabai wrote:
| > It's faster at AI than an Nvidia RTX4090, because 96GB of the
| 128GB can be allocated to the GPU memory space
|
| I love AMD's Ryzen chips and will recommend their laptops over
| an Nvidia model all day. However, this is a pretty facetious
| comparison that falls apart when you normalize the memory. Any
| chip can be memory bottlenecked, and if we take away that
| arbitrary precondition the Strix Halo gets trounced in terms of
| compute capacity. You can look at the TDP of either chip and
| surmise this pretty easily.
| dist-epoch wrote:
| > However, this is a pretty facetious comparison that falls
| apart when you normalize the memory
|
| Why would you normalize though? You can't buy a 96 GB
| RTX4090. So it's fair to compare the whole deal, slowish APU
| with large RAM versus very fast GPU with limited RAM.
| andrewstuart wrote:
| " AMD also claims its Strix Halo APUs can deliver 2.2x more
| tokens per second than the RTX 4090 when running the Llama
| 70B LLM (Large Language Model) at 1/6th the TDP (75W)."
|
| https://www.tomshardware.com/pc-components/cpus/amd-
| slides-c...
|
| You could argue it's invalid claim because it's from AMD not
| independent.
| dr_kiszonka wrote:
| It looks like it will be available in the Framework Desktop! I
| would love to see it in a more budget mini PC at some point
| from another company. (Framework is great but not in my price
| range.)
| grg0 wrote:
| The issue is that programming a discrete GPU feels like
| programming a printer over a COM port, just with higher
| bandwidths. It's an entirely moronic programming model to be
| using in 2025.
|
| - You need to compile shader source/bytecode at runtime; you
| can't just "run" a program.
|
| - On NUMA/discrete, the GPU cannot just manipulate the data
| structures the CPU already has; gotta copy the whole thing over.
| And you better design an algorithm that does not require
| immediate synchronization between the two.
|
| - You need to synchronize data access between CPU-GPU and GPU
| workloads.
|
| - You need to deal with bad and confusing APIs because there is
| no standardization of the underlying hardware.
|
| - You need to deal with a combinatorial turd explosion of
| configurations. HW vendors want to protect their turd, so drivers
| and specs are behind fairly tight gates. OS vendors also want to
| protect their turd and refuse even the software API standard
| altogether. And then the tooling also sucks.
|
| What I would like is a CPU with a highly parallel array of
| "worker cores" all addressing the same memory and speaking the
| same goddamn language that the CPU does. But maybe that is an
| inherently crappy architecture for reasons that are beyond my
| basic hardware knowledge.
___________________________________________________________________
(page generated 2025-03-21 23:00 UTC)