[HN Gopher] Running PyTorch on the M1 GPU
___________________________________________________________________
Running PyTorch on the M1 GPU
Author : my123
Score : 88 points
Date : 2022-05-21 09:16 UTC (13 hours ago)
(HTM) web link (sebastianraschka.com)
(TXT) w3m dump (sebastianraschka.com)
| machinekob wrote:
| MPS (Metal backend for PyTorch) is pretty poor integrated right
| now most of my custom models are crashing and pure Swift +
| MPSGraph versions is working 3-10x faster then PyTorch. So I'm
| pretty sure there is A LOT of optimizing and bug fixing before we
| can even consider PyTorch on apple devices (and this is ofc. 0.1
| or smth like that so maybe we'll get something working fine in
| next few years).
|
| Also it'll be cool to compare Tensorflow vs PyTorch on metal
| device :)
| danieldk wrote:
| _MPS (Metal backend for PyTorch) is pretty poor integrated
| right now most of my custom models are crashing and pure Swift
| + MPSGraph versions is working 3-10x faster then PyTorch. So I
| 'm pretty sure there is A LOT of optimizing and bug fixing
| before we can even consider PyTorch on apple devices_
|
| Also, even for ops that do not crash, it often returns garbage.
| I built PyTorch from git this morning: >>>
| torch.arange(10, device="mps") tensor([0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0], device='mps:0') >>> torch.ones(10,
| device="mps").type(torch.int32) tensor([1065353216,
| 1065353216, 1065353216, 1065353216, 1065353216, 1065353216,
| 1065353216, 1065353216, 1065353216, 1065353216],
| device='mps:0', dtype=torch.int32)
|
| Many bugs will probably be squashed over the coming weeks, so I
| am still very exited about MPS support. I have done some
| preliminary benchmarks with a spaCy transformer model and the
| speedup was 2.55x on an M1 Pro. Which is quite nice because
| transformers were already very fast on M1 Macs, thanks to the
| AMX units (PyTorch links against Accelerate, so the AMX units
| are used for matrix multiplication). If they can squeeze more
| performance out of M1 GPUs in the future, it will be a very
| nice speedup.
| machinekob wrote:
| Weeks :? Looking at every other project from Apple that try
| to integrate with some opensource lib it'll be months/years
| if they didn't drop support for it before that :P
|
| Did you get only 2.55x on BERT/other transformer vs CPU
| version?
| danieldk wrote:
| I think in this case the PyTorch team also involved. They
| have already fixed some annoying bugs in the last two days
| (like matrix multiplication often failing because of buffer
| size mismatches).
|
| As for BERT inference performance, I am not sure what kind
| of speedups you are expecting. The M1 Pro gives me 2.6
| TFLOPs in single precision matrix multiplication of 768x768
| matrices. M1 Pro GPU performance is supposed to be 5.3
| TFLOPS (not sure, I haven't benchmarked it).
| machinekob wrote:
| Ahh nvm I was thinking about m1 max (my brain is damaged
| by the m1 naming and i didnt saw information that you are
| using m1 pro gpu)
| danieldk wrote:
| Right, the Max should make a much bigger difference,
| since it has the same number of AMX units as the Pro, but
| double the GPU cores.
| [deleted]
| neogodless wrote:
| Related post:
|
| https://news.ycombinator.com/item?id=31424048
|
| Accelerated PyTorch Training on M1 Mac (pytorch.org)
|
| 436 points by tgymnich 2 days ago 143 comments
| londons_explore wrote:
| What ML are people actually wanting to do on M1 laptops?
|
| I was under the impression that outside toy problems, you can't
| really get decent results unless you have a TPU or fleet of GPU's
| for training. If you just train on a laptop you're shooting
| yourself in the foot compared to someone who gets the same job
| done 10x faster in the cloud (usually with free credits).
|
| Obviously inference is still possible on a laptop for many
| models... but not many 'hacker' tasks involve only inference.
| danieldk wrote:
| _What ML are people actually wanting to do on M1 laptops?_
|
| (1) development - quickly train smaller networks locally to
| validate the code. Then offload to a big GPU machine. (2)
| Smaller convolutional networks and RNNs may be below SOTA, but
| can still provide a good performance-accuracy trade-off. You
| could already train these networks pretty fast on an M1 thanks
| to AMX, but his makes training such networks even faster.
| visionscaper wrote:
| I'm not using an M1 machine yet, but I think is extremely
| useful to be able to do some small scale experiments for
| testing and debugging purposes locally before starting a full-
| scale training job in the cloud.
|
| Further, in general, I'm really charmed by the potential of
| having unified memory; the idea you can test some batch
| training iterations with a batch size that fills almost all
| 128MB is a unique capability!
|
| Last, a large amount of unified memory allows to do inference
| and "prompt engineering" with very large models, locally. E.g.
| Using GPT-J like models (6B parameters).
| singularity2001 wrote:
| Ideally fine-tuning foundation models. Training them costs
| several millions, so out of scope no matter which GPU you have.
| Tenoke wrote:
| >I was under the impression that outside toy problems, you
| can't really get decent results unless you have a TPU or fleet
| of GPU's for training.
|
| This is most certainly not the case. It's impossible to achieve
| SOTA on most benchmarks but there's a lot more ML and data
| analysis done on smaller datasets where you don't particularly
| need a fleet of GPUs.
| nl wrote:
| I've fine tuned transformers on CPUs before for weird reasons.
| Sometimes the computer you have is what you make do with.
| arecurrence wrote:
| There's plenty of training related stuff that 10 TFlops is
| sufficient for. Fine tuning models for example is a great use
| case for a high ram yet relatively low TFlops environment.
|
| I don't know how much you've done with running ml jobs on cloud
| boxes but it can be very annoying to work with them when you
| are actively iterating. Especially if you are doing a lot of
| work outside the cloud box environment. Even turn key consumer
| products like Colab can fall over if you accidentally press
| "Back" in your browser... I once lost many hours of work
| because I inadvertently swiped left :)
|
| Apple will continue improving their performance in this space
| and once jobs that run for hours can be run overnight... I
| suspect people will often choose to run overnight simply for
| the ergonomics benefit.
| shimmeringleaf wrote:
| Totally, performance for actual useful training etc. is at
| least a couple of hardware generations away. This is super
| useful however for people like me who like to use an m1 Mac as
| their daily driver and don't always want to have to fire up a
| remote rig just to sketch a little. Also being able to do some
| local inference can be great for when building a web GUI, etc.
|
| So absolutely not mature, but having PyTorch working better
| also on macs is only positive imho.
| andybak wrote:
| The whole Diffusion text to image thing which is hugely popular
| right now doesn't involve any training. Same for text
| generation where you download pre-trained models. I'm using a
| laptop with a 3080 and getting much faster results than I
| typically get from a paid Colab account (not Pro mind you).
| natly wrote:
| Which repo/tool are you using for local text to image
| diffusion if you don't mind sharing? :)
| andybak wrote:
| Not at all. I discovered that Visions of Chaos (originally
| developed for fractals etc) has extensive support for
| installing and running ML models
|
| https://softology.pro/tutorials/tensorflow/tensorflow.htm
|
| It handles (most) of the installation issues and gives you
| a GUI. You can however bypass the GUI. Every time you run a
| command the console shows you what the command line looks
| like so copy/paste into a terminal and away you go.
| lvl102 wrote:
| Can someone explain to me why M1 Ultra doesn't scale ratably
| compared to RTX? It seems it should perform at about 1/2 of 3090.
| Is it because Apple locked down what you can do with the GPU?
| brrrrrm wrote:
| If the tensor cores are used, the 3090 can hit 300Tflops. I
| don't think the M1 can even come close to that (21Tflops for
| the ultra)
| lvl102 wrote:
| Makes sense so it's basically 7x more just in terms of raw
| power. I was wrongly under the impression that Apple was
| limiting ML capacity.
| Iwan-Zotow wrote:
| 1/2? 1/2?!?
|
| make it at least 1/10
| SemanticStrengh wrote:
| Does the M1 support openCL/SYCL on macos?
| olliej wrote:
| Apple aggressively pushed openCL when it was new, and NVIDIA
| and AMD aggressively pushed their platform specific CUDA APIs,
| so all anyone did was complain about apple using openCL when no
| vendors supported it.
|
| Apple had (runtime and ABI) stable openCL built into macOS and
| iOS by default and on desktop devs kept requiring you install
| NVIDIAs or AMDs CUDA systems, and on iOS everyone found that
| openCL was just as clunky to use as openGL.
|
| Apple made proprietary APIs that were more usable and better
| interacted with the hardware, and everyone just used those
| instead, until Apple deprecated openCL and openGL as they
| merely represent a pile of engineering work and support that is
| essentially unused.
| my123 wrote:
| OpenCL 1.2* with very few extensions, pretty much stuck in the
| past. Don't even think about using unified memory from there.
|
| SYCL: due to the above, not on the GPU. Just on the CPU.
|
| * x86_64 apps also get to see the OpenCL CPU backend in
| addition to the GPU one. However, arm64 apps only see the GPU.
| SemanticStrengh wrote:
| Apple is an irrelevant company at this point
| pjmlp wrote:
| Maybe for those that buy Apple as shinny UNIX instead of
| supporting Linux OEMs, for others there are plenty of
| market opportunities.
| atty wrote:
| On the contrary, I believe MacBooks have continued to eat
| general market share since they switched to Apple Silicon,
| and they are by far the best built UNIX-based laptop you
| can get, and have by far the best battery life. There's a
| reason so many devs (and scientists, like myself) use them.
| olliej wrote:
| They're the fourth largest laptop vendor, making 9% market
| share, up from 7% the year prior [1]
|
| Apple is #2 world wide for phones at 18% [2]
|
| Apple is #1 for phones in the US with 51% share [3]
|
| Apple is #2 for phones in the EU at 23% share [4]
|
| Apple's laptop seem to wipe the floor in power usage, while
| not getting anyway more in gaming land than any prior
| generation [5]
|
| Apple's revenue is the 3rd highest in the world which also
| doesn't seem particularly irrelevant.
|
| You can hate Apple all you want, but going "I don't like
| apple, therefore the company is irrelevant" is a dumbass
| approach. Hell what will android get without finding out
| what's in iOS?
|
| [1] https://9to5mac.com/2022/04/11/mac-market-bucks-trend-
| with-c...
|
| [2] https://www.canalys.com/newsroom/global-smartphone-
| market-Q1...
|
| [3] https://www.imore.com/apple-dominates-us-smartphone-
| market-5... (This is iMore so take with a grain of salt)
|
| [4] https://www.appleworld.today/2022/02/28/apples-iphone-
| has-23...
|
| [5] https://www.pcmag.com/news/intel-core-i9-vs-
| apple-m1-max-whi...
| fartcannon wrote:
| Too bad there's a requirement for metal. I had tensorflow running
| on a raspberry pi 3 a few years ago doing inference for a chat
| bot.
| Raymonf wrote:
| Why wouldn't GPU support for the M1 GPU require Metal? You
| should be able to run PyTorch on the CPU of a Pi?
| fartcannon wrote:
| Metal is an unnecessary complication.
| olliej wrote:
| How so? NVIDIA GPUs have their own APIs, AMD does, Direct3D
| is its own. The Apple platform acceleration APIs are Metal.
| The MS acceleration APIs are all generally Direct3D, etc.
|
| The fact the _you want_ Apple to support other platforms
| doesn 't mean that they're wrong for not doing so. They
| same arguments to require Apple to support your preferred
| API also applies to MS, NVIDIA, AMD, etc and the same
| arguments to not do that apply them all as well.
| Raymonf wrote:
| Then what API are you going to use on macOS?
| fartcannon wrote:
| You're not permitted to use anything else, thus: too bad
| it has a requirement for metal.
___________________________________________________________________
(page generated 2022-05-21 23:02 UTC)