[HN Gopher] CUDA-l2: Surpassing cuBLAS performance for matrix mu...
___________________________________________________________________
CUDA-l2: Surpassing cuBLAS performance for matrix multiplication
through RL
Author : dzign
Score : 126 points
Date : 2025-12-04 21:04 UTC (1 days ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| stonogo wrote:
| Am I reading this wrong, or does this only support FP16 inputs,
| and compares its performance against an FP32 solver?
| j2kun wrote:
| They claim the algorithm "discovered" the new techniques, but the
| methods described in section 5 do not seem all that novel to me.
| It smells like it could be "laundering" the literature [1] and
| reshuffling existing techniques. This is not inherently a bad
| thing, but I would hope that if it is borrowing existing
| techniques, the appropriate citation would eventually make it
| into this paper.
|
| [1]: https://www.argmin.net/p/lore-laundering-machines
| AlexCoventry wrote:
| In the future, we will all be Jurgen Schmidhuber. :-)
| hedgehog wrote:
| I hate to break it to you but the original work on that topic
| was by Schmidhuber & Schmidhuber back in 1963.
| alyxya wrote:
| There generally aren't new techniques when optimizing something
| ubiquitous. Instead, there are a lot of ways to apply existing
| techniques to create new and better results. Most ideas are
| built on top of the same foundational principles.
| slashdave wrote:
| I am not sure about that. However, what is clear is that if
| there is a new technique, it will not be found by this LLM.
| CapsAdmin wrote:
| It's generally true, isn't it? Otherwise we'd have ground
| breaking discoveries every day about some new and fastest
| way to do X.
|
| The way I see it, mathematicians have been trying (and
| somewhat succeeding every 5~ years) to prove faster ways to
| do matrix multiplications since the 1970s. But this is only
| in theory.
|
| If you want to implement the theory, you suddenly have many
| variables you need to take care of such as memory speed,
| cpu instructions, bit precision, etc. So in practice, an
| actual implementation of some theory likely have more room
| to improve. It is also likely that LLM's can help figure
| out how to write a more optimal implementation.
| josephg wrote:
| Yes. And there's still lots of places where you can get
| significant speed ups by simply applying those old techniques
| in a new domain or a novel way. The difference between a
| naive implementation of an algorithm and an optimised one is
| often many orders of magnitude. Look at automerge - which
| went from taking 30 seconds on a simple example to tens of
| milliseconds.
|
| I think about this regularly when I compile C++ or rust using
| llvm. It's an excellent compiler backend. It produces really
| good code. But it is incredibly slow, and for no good
| technical reason. Plenty of other similar compilers run
| circles around it.
|
| Imagine an llvm rewrite by the people who made V8, or chrome
| or the unreal engine. Or the guy who made luajit or the Go
| compiler team. I'd be shocked if we didn't see an order of
| magnitude speed up overnight. They'd need some leeway to
| redesign llvm IR of course. And it would take years to port
| all of llvm's existing optimisations. But my computer can
| retire billions of operations per second. And render
| cyberpunk at 60fps. It shouldn't take seconds of cpu time to
| compile a small program.
| Q6T46nT668w6i3m wrote:
| You're not kidding. I just looked. There isn't anything novel
| in that section. I assumed from the description they found
| novel methods but this is standard GPU Gems advice.
| alyxya wrote:
| The chart confused me because I expected to see performance
| numbers of CUDA-L2 compared to the others, but instead it shows a
| chart showing the speedup percentage of CUDA-L2 over the others.
| In some sense, the bar chart effectively inverts the performance
| of torch.matmul and cuBLAS with how much percentage it shows. 0%
| on the bar chart would only mean equal performance.
| konradha wrote:
| I've been trying my hand at RL envs for various sparse matrix
| algorithms in CUDA. It's easy to generate code that "looks good",
| "novel" and "fast". Escaping the distribution and actually
| creating novel sequences of instructions or even patterns (has
| any model come with something as useful as fan-in/fan-out or
| double buffering patterns that's now ubiquituous?) seems
| difficult to say the least.
| roflmaostc wrote:
| > Q: What if I need matrix dimensions (M, N, K) not found in your
| configurations? >A: 1. You can find the nearest neighbor
| configuration (larger than yours) and pad with zeros. 2. Feel
| free to post your dimensions on GitHub issues. We are happy to
| release kernels for your configuration.
|
| Lol, this will be potentially much slower than using the general
| matmul kernel.
|
| However, I like this kind of research because it really exploits
| specific hardware configurations and makes it measurable faster
| (unlike some theoretical matmul improvements). Code
| specialization is cheap, and if it saves in the order of a few %,
| it quickly reimburses its price, especially for important things
| like matmul.
___________________________________________________________________
(page generated 2025-12-05 23:02 UTC)