Post ASDn1Ap6rBlaFahbOK by marcoshuerta@vmst.io
(DIR) More posts by marcoshuerta@vmst.io
(DIR) Post #AS8qDz27KeQhWKmpqS by simon@fedi.simonwillison.net
2023-01-29T17:02:42Z
0 likes, 0 repeats
Has anyone had any success running any of the open source-ish large language models on an M1/M2 Mac?If so, how did you do it?
(DIR) Post #AS8qbw0MF8Np4uNtlw by llimllib@vis.social
2023-01-29T17:06:34Z
0 likes, 0 repeats
@simon I put reasonably serious effort into trying, but the extremely incomplete state of the torch backend for metal (MPS) [1] stymied me. ATM it seems to me that the arm macs are bad platforms for LLM work, and that seems unlikely to change any time soon[1]: https://github.com/pytorch/pytorch/issues/77764
(DIR) Post #AS8rBkJxA38vGYctWq by blaine@mastodon.social
2023-01-29T17:11:12Z
0 likes, 0 repeats
@simon @mattb shared a link from some Apple peeps who had done this, but I don't have the link handy right now; I can dig it up when I'm back at a computer later if you don't find anything sooner (I know it's been done, though!)
(DIR) Post #AS8rZ3DX4RbIX5jfdY by simon@fedi.simonwillison.net
2023-01-29T17:14:12Z
0 likes, 0 repeats
It looks like the answer may involve PyTorch and these unofficial builds for the M1 https://github.com/mlverse/libtorch-mac-m1/releases
(DIR) Post #AS8rnmAEkwS9zGDQky by ab@sigmoid.social
2023-01-29T17:16:12Z
0 likes, 0 repeats
@simon yep, https://diffusionbee.com/
(DIR) Post #AS8sIkQFHY5ZYuTKGu by bmann@social.coop
2023-01-29T17:16:32Z
0 likes, 0 repeats
@blaine @simon @mattb paper from Apple itself on Stable Diffusion https://machinelearning.apple.com/research/stable-diffusion-coreml-apple-siliconThere are several howtos and packages. I know you’re probably asking for ChatGPT~ish stuff rather than images. #OpenAssistant appears to be a good OSS direction https://github.com/LAION-AI/Open-Assistant
(DIR) Post #AS8sIkxZHekBEG3wDA by simon@fedi.simonwillison.net
2023-01-29T17:23:43Z
0 likes, 0 repeats
@bmann @blaine @mattb yeah running Stable Diffusion feels relatively straight-forward at this point - but language models are a whole lot larger and more challengingI'm not expecting to be able to run a good one, but anything at all which can generate some text would make an interesting starting point
(DIR) Post #AS8sf9DVmEDqpeCGlU by simon@fedi.simonwillison.net
2023-01-29T17:26:14Z
0 likes, 0 repeats
@ab turns out the image models are easy! I want to run a language model, basically a much less impressive version of GPT-3
(DIR) Post #AS8sqEFNPouxcNLxYG by gcampax@mastodon.social
2023-01-29T17:31:25Z
0 likes, 0 repeats
@simon define large- a BERT sized or maybe some of the smaller T5s will run no problem, but a GPT3 sized model needs multiple high end GPUs to run...
(DIR) Post #AS8tAlbN73uErER04W by simon@fedi.simonwillison.net
2023-01-29T17:35:18Z
0 likes, 0 repeats
@gcampax yeah I'm not expecting to be able to run anything slow to a GPT-3 - but running any language model at all on an M2 would be super interesting to meHave you run a BERT or T5 on one? Usable instructions seem difficult to locate
(DIR) Post #AS8x8l5USAsKV4ott2 by bmann@social.coop
2023-01-29T18:09:58Z
0 likes, 0 repeats
@simon @blaine @mattb I’m keeping an eye on #OpenAssistantThe LAION group behind it is the German charity that gathered the images for the OSS stable diffusion stuff.
(DIR) Post #AS90SrAxpJiZ2LehRw by ab@sigmoid.social
2023-01-29T18:32:26Z
0 likes, 0 repeats
@simon sorry, misread your original post!Makes me think that the next “Mac Pro" should be a cloud GPU service
(DIR) Post #AS913SXQOMstL4cAyW by gcampax@mastodon.social
2023-01-29T18:54:36Z
0 likes, 0 repeats
@simon my colleague runs our models (smaller BERTs using pytorch+huggingface) on his M1 and yeah it used to be a problem, especially with poetry, but these days as I understand it the most recent version of pytorch will naturally install the M1 backend with pip. Don't fiddle with conda and use native brew. Not sure about M2 but I imagine it will be similar.
(DIR) Post #AS9ucCeOwqOlgASSLw by mkasu@sigmoid.social
2023-01-30T05:26:05Z
0 likes, 0 repeats
@simon PyTorch 1.13 and newer comes with official M1 support https://pytorch.org/blog/PyTorch-1.13-release/ So, getting PyTorch to run is easy as installing the official packages via PIP (However, the model code needs to be adjusted from CUDA to MPS target device. Also, there might be incompatabilities etc. as the computational feature set isn’t the same I think)
(DIR) Post #ASBYBzigap6jN0TnNo by simon@fedi.simonwillison.net
2023-01-31T00:24:41Z
0 likes, 0 repeats
Relevant: John Lam reports successfully running the gtr-t5-large model for calculating embeddings on an M1: https://twitter.com/john_lam/status/1620209726024978433That model on Hugging Face: https://huggingface.co/sentence-transformers/gtr-t5-large
(DIR) Post #ASDVNWzxIFIsDkHpSq by simon@fedi.simonwillison.net
2023-01-31T22:59:53Z
0 likes, 0 repeats
I got this to work! https://til.simonwillison.net/python/gtr-t5-large
(DIR) Post #ASDn1Ap6rBlaFahbOK by marcoshuerta@vmst.io
2023-02-01T02:19:46Z
0 likes, 0 repeats
@simon when I was saving the sentence-transformer vectors for my semantic emoji finder I would wrap the numpy array data in a pandas frame, and save it to a parquet file. Come to think of it, I see that the repository is still littered with ill-named parquet files. 😬(I really hope we get more progress on metal shaders support for pytorch. Things like sentence-transformer could benefit from the Apple silicon GPU.)