[HN Gopher] Exploring the scalable matrix extension of the Apple...
___________________________________________________________________
Exploring the scalable matrix extension of the Apple M4 processor
Author : gok
Score : 159 points
Date : 2024-09-10 15:03 UTC (3 days ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| freeqaz wrote:
| Any comparison with how much faster this is compared with the
| previous way of doing things on the CPU?
| svnt wrote:
| Based on my understanding from the description, it is ~8x
| faster (250 GFLOPS) for vector ops (vs. SVE mode at 31 GFLOPS
| which is CPU-ish) and 60-100 times faster (e.g. 2005 GFLOPS)
| for matrix multiplication for single-precision values.
| jandrese wrote:
| That's alright, but not mindblowing. How does it compare to
| doing the same work on a GPU? Is there a particular set of
| tasks that GPUs struggle with that would be well suited for
| this? Or is this more a fig leaf over lousy GPU compute
| support in Apple land?
| bee_rider wrote:
| Apple should mostly care about power-efficient inference I
| think, right? Not training. Spinning up a GPU seems like
| something to avoid.
|
| I mean, I wonder how this thing compares to a gemm using
| all the cores in a cpu cluster. They might be ok with not
| even meeting that performance, if the accelerator can not
| hog all the cores and power.
|
| At least that's what my uninformed gut says. The workload
| for these things is like: little AI enhancements inside
| conventional apps, I think.
| lxgr wrote:
| > Spinning up a GPU seems like something to avoid.
|
| You can do inference on GPUs as well, and for anything
| other than very small/lightweight models, such as noise
| cancellation or maybe speech recognition, it's probably
| worth the initial overhead.
|
| I believe CoreML already splits workloads between CPU,
| NPU, and GPU as appropriate.
| huijzer wrote:
| 60 times faster could mean 2 minutes instead of 2 hours, or
| 2 seconds instead of 2 minutes. How is that not mind
| blowing, or at least very useful (for specific uses)?
| jandrese wrote:
| Compared to 600 or 6000 times faster on a GPU though?
| ein0p wrote:
| I'm not sure why they added this feature. All Apple SoCs have far
| more energy efficient compute than the CPU. This would only make
| sense for really tiny models which need extremely quick forward
| pass. For such models the overhead of a GPU or Neural Engine
| kernel launch would be quite noticeable. But for those the old
| NEON was already OK, and if not, there also is a dedicated matrix
| unit there called AMX. Seems kinda random to me.
| adrian_b wrote:
| This replaces AMX, it is its successor.
|
| The older Apple CPUs implemented a custom form of AMX that was
| not standardized by Arm.
|
| Presumably as a result of cooperation with Apple, the Arm ISA
| now includes a set of instructions with the same purpose like
| the original Apple AMX.
|
| The newer Apple CPUs have been updated to use the standard Arm
| ISA, instead of their older proprietary ISA.
|
| In the Apple CPUs, the former AMX and the current SME provide a
| much higher throughput than the CPU cores, even if lower than
| the GPU, and a much lower latency than the GPU, even if higher
| than the CPU cores.
|
| AMX/SME is implemented as a separate accelerator, distinct from
| the CPU cores, because this saves power and area in comparison
| with implementing such instructions in each CPU core. The Apple
| CPUs do not attempt to compete in high-performance computing
| applications, so the extra throughput provided by a separate
| shared matrix operation accelerator is good enough for them.
| saagarjha wrote:
| This has both actually.
| Archit3ch wrote:
| Dedicated FP64 is great for real-time audio processing. Like an
| included DSP chip.
| phkahler wrote:
| Isn't FP32 sufficient for audio processing? Even though we
| have 24bit DACs and ADCs these days I feel like 16bit was
| really good enough. FP32 with 24bit mantissa should avoid
| rounding errors at the 16bit level right?
| Archit3ch wrote:
| It depends on the application.
|
| 16bit is enough for representation.
|
| 24bit is enough for recording (some leeway because
| recording levels won't be ideal).
|
| FP32 for processing with simple effects (e.g. mixer, some
| EQs). If that's enough for your needs, you can SIMD/GPU to
| your heart's content.
|
| FP64 for high Q filters, phasors, LU decompositions.
| GeekyBear wrote:
| Matrix multiplication is very commonly used in science and
| engineering, not just machine learning.
|
| The neural engine is optimized for machine learning use cases.
|
| This standardized successor to AMX is more general purpose than
| the neural engine and has much improved matrix multiplication
| performance vs NEON.
|
| As a bonus, since this is no longer just an experimental
| implementation of a matrix unit, you get documented access to
| the new ARM standardized low level instruction set.
| brigade wrote:
| The neural engine by design cannot handle all possible kernels,
| and the GPU is significantly slower for integer math, and
| cannot do fp64. Then for the iPhone SoCs with 4 or 5 core GPUs,
| the GPU is a bit slower for fp16 and fp32 too.
| lxgr wrote:
| > if not, there also is a dedicated matrix unit there called
| AMX
|
| This seems to be the successor to AMX.
| dividuum wrote:
| > Although Apple has included a matrix accelerator in its devices
| since 2019, it used a proprietary instruction set inaccessible to
| developers, who officially could only use Apple-provided
| numerical libraries.
|
| How does that work? Does the hardware throw some kind of fault
| when using those instructions? Or are they merely undocumented
| and you could use them if you figure out how they work? I guess
| the second, as hinted by the "officially"?
| my123 wrote:
| Merely undocumented
| jonstewart wrote:
| Peter Cawley has a good write-up on the undocumented M1/M2/M3
| AMX instructions: https://github.com/corsix/amx
| bee_rider wrote:
| As others have said, just undocumented.
|
| IIRC there was a BLIS fork that used AMX instructions. I think
| it was unofficial though(?). It is hard to do science without
| properly documented tools.
| nxobject wrote:
| If Apple's going for one SME accelerator per base M4 chiplet,
| it'll be interesting to see how to program scalably for
| Pro/Max/Ultra variants.
| wtallis wrote:
| You should be thinking in terms of CPU clusters, not chiplets.
| The Ultra is the only one with multiple chiplets, but all of
| their processors have multiple CPU clusters, and so far it's
| one AMX/SME per cluster.
| bee_rider wrote:
| I guess the the CPU/cluster and cluster/chiplet ratios change
| from generation to generation?
| wtallis wrote:
| They're not constant even within a generation. The M3, M3
| Pro, and M3 Max are each monolithic SoCs of different sizes
| (no chiplets) with different CPU cluster configurations,
| and the phone chip of the same generation is yet another
| configuration.
| nxobject wrote:
| Ah, thank you! That's the right word. They're on the same
| die, no, so "chiplet" isn't the appropriate word?
| softwaredoug wrote:
| I just wish they'd make native tensorflow installation actually
| work without a million apple silicon specific exceptions :)
| TheFuzzball wrote:
| They will, just in time for everyone to have switched to
| pytorch!
| kjkjadksj wrote:
| I wish they made computers that ran software like games again.
| Seems like the last few iterations they've been working hard on
| making computers that are able to run ai models a little faster.
| Are people really asking for that? I would think far more people
| would like to play a video game over rolling their own matrix
| multiplication, but I guess that's why they pay the people at
| apple the big bucks because they must know best.
| jwells89 wrote:
| Overall, GPU strength is the best it's ever been in portable
| Apple devices by a significant margin. The problem isn't the
| hardware, it's that game developers are reticent to support
| anything that's not x86 Windows+DirectX or one of the consoles.
|
| It's often said that macOS/iOS supporting Vulkan would help and
| while I think that's true to an extent, native Vulkan support
| is still rare enough that it's not going to change all that
| much in terms of ease of porting. It might improve things on
| the front of running games through WINE (DirectX - Vulkan
| translation), but unless developers produce ARM builds of their
| games there's always going to be the overhead of being run
| through an x86 translator, which varies depending on how CPU
| heavy the game is.
| samatman wrote:
| Are you under the impression that fast matrix operations in the
| CPU are useless for,, games?
|
| Where did you get that idea?
| naming_the_user wrote:
| Modern Apple Silicon based laptops have fantastic graphics
| performance, manufacturers just aren't that interested in
| supporting them.
|
| It's probably a bit of a chicken and egg thing at this point,
| plus the fact that most "serious" gamers are going to have
| desktop PC's anyway.
| Detrytus wrote:
| I thought one of the reasons to bring Apple Silicon to Mac
| was that all the iPhone games can now be easily ported?
| lxgr wrote:
| > manufacturers just aren't that interested in supporting
| them
|
| AAA games are starting to show up on Steam for macOS these
| days. Baldur's Gate 3 runs pretty well, for example!
|
| The real shame is that some older indie games are
| disappearing just as easily, given Apple's deprecation
| strategy - while Microsoft basically never breaks backwards
| compatibility, Apple recently cut off 32 bit games (killing
| about half my Steam library), and presumably Intel-only
| binaries are next.
| wtallis wrote:
| Apple dropped support for 32-bit Mac applications five
| years ago; recent only by comparison to Microsoft's
| theoretical backwards compatibility. Apple dropped support
| for 32-bit Mac hardware, firmware, and drivers in 2012, so
| there was a period of seven years where game developers had
| every reason to make their Mac releases 64-bit, but to a
| disappointingly large degree they didn't.
|
| This was probably due in large part to a lack of pressure
| on the Windows side. It was absolutely absurd that even a
| big budget (and memory-hungry) game like Skyrim was
| released in 2011 as a 32-bit only game, and didn't get a
| 64-bit release until 2016.
|
| I didn't enjoy macOS killing compatibility with so much of
| my Steam library either, but I do at least respect that
| Apple had some solid reasons, and save some of my ire for
| the game devs that shipped outdated binaries.
| aseipp wrote:
| You can spend a small amount of die space on something that
| will yield 10x performance benefits for some things, and you
| can spend a lot of die space on something that will only yield
| a general 5% improvement. Which you choose depends on a lot of
| factors. In other words, the relationship between the "things
| on the chip" and general performance, or specific application
| performance, is not a strictly linear relationship.
|
| The 20 series Nvidia GPUs with RTX were a good example. RT
| cores were added and took up significant die space, people said
| "why not more CUDA cores", but given the design of consumer
| GPUs it's extremely unlikely that just replacing those with
| more CUDA cores would have had a proportional uplift. In
| Nvidia's case, they realized RT cores were a better bet and
| served their customer bases (industrial graphics, gaming)
| better than just more raw numbers.
|
| As it stands, specialization like this is a key element of new
| designs on leading edge processes. You're going to see more of
| it, not less.
|
| > I guess that's why they pay the people at apple the big bucks
| because they must know best.
|
| Well I don't know about "best", they almost certainly know
| ~infinitely more about their customers and workloads than
| random people like us do, I can at least say that much.
| wmf wrote:
| _The 20 series Nvidia GPUs with RTX were a good example. RT
| cores were added and took up significant die space, people
| said "why not more CUDA cores"_
|
| Or they could have had the same number of CUDA cores without
| RT at a lower price (the fabled "1180")...
| fragmede wrote:
| They are! The graphics for video games are just a series of
| matrix multiplications. Before it can get shown to the screen,
| the graphics are a bunch of triangles, represented by matrices,
| and in order to do anything in game, those matrices need to be
| multiplied in order to move them around in 3d space, before
| getting rendered out to the screen. Making computers better at
| matrix math means better rendering for video games.
| TOMDM wrote:
| Apple literally marketed the new iPhone running Death Stranding
| lxgr wrote:
| Are you implying that recent Apple SoCs _can 't_ run games?
|
| While there's the ML-centric "Neural Engine", the GPU really
| isn't stagnating by any means: Just in the iPhone 16
| presentation this week, ray tracing and a 20% faster GPU were
| among the headline features. Gaming got its own section in the
| video presentation!
|
| The fastest GPU I own is in my Mac; the second fastest is in my
| iPhone. My dedicated (last-gen) game consoles are a distant
| third and forth, respectively.
| brcmthrowaway wrote:
| I'm dim, whats the difference between SVE and SME?
| mrmuagi wrote:
| Vector vs Matrices. Higher dimensional.
___________________________________________________________________
(page generated 2024-09-13 23:01 UTC)