Post AcJXMb4EurM37XOPcu by saurabhs@mas.to
(DIR) More posts by saurabhs@mas.to
(DIR) Post #AcJH0GbQixMQ0sFPg8 by simon@fedi.simonwillison.net
2023-11-29T20:30:23Z
0 likes, 2 repeats
Here's an incredible new way to run LLMs on your own machine: llamafile - https://hacks.mozilla.org/2023/11/introducing-llamafile/It bundles an LLM with the code needed to run it in a single binary using DEEP magic (Cosmopolitan Libc) such that the same binary works on 6 different operating systemsBest part: it works with LLaVA multi-modal... so you can a 4GB file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/llamafile-server-0.1-llava-v1.5-7b-q4 and:chmod 755 llamafile-server-0.1-llava-v1.5-7b-q4./llamafile-server-0.1-llava-v1.5-7b-qVisit http://127.0.0.1:8080/And now:
(DIR) Post #AcJJRZnre2KhI1DIy8 by simon@fedi.simonwillison.net
2023-11-29T20:58:06Z
0 likes, 0 repeats
Wrote more about this on my blog:**llamafile is the new best way to run a LLM on your own computer**https://simonwillison.net/2023/Nov/29/llamafile/
(DIR) Post #AcJKGyaxEGfKwsgUoS by adr@mastodon.social
2023-11-29T21:06:59Z
0 likes, 0 repeats
@simon Whoa. This is REALLY NEAT.
(DIR) Post #AcJKktJ1SVWxERYrtw by gwire@mastodon.social
2023-11-29T21:12:29Z
0 likes, 0 repeats
@simon I'd suggest something like “curl -L -O https:…” instead of wget, since wget doesn’t ship with stock macOS.
(DIR) Post #AcJL0dUS1qauoWPmDY by jonsjava@mastodon.world
2023-11-29T21:12:41Z
0 likes, 0 repeats
@simon Neat idea, but sadly, when I try that, I get this error:zsh: exec format error: ./llamafile-server-0.1-llava-v1.5-7b-q4When I run file on it:llamafile-server-0.1-llava-v1.5-7b-q4: DOS/MBR boot sector; partition 1 : ID=0x7f, active, start-CHS (0x0,0,1), end-CHS (0x3ff,255,63), startsector 0, 4294967295 sectorsRunning on Debian derivative (Pop! OS)EDIT: need to run in BASH
(DIR) Post #AcJNDA5eqdW7Iwv0JE by mergesort@macaw.social
2023-11-29T21:40:26Z
0 likes, 0 repeats
@simon Which M2 do you have? I'm curious because the profile between and M2 Air and an M2 Mac Pro are very different. 😅
(DIR) Post #AcJNfXG6Im2NGiKgIS by simon@fedi.simonwillison.net
2023-11-29T21:45:35Z
0 likes, 0 repeats
@mergesort I'm on a M2 MacBook Pro with 64GB of RAM, but LLaVA only uses 1.2GB, presumably because it offloads a lot of the model to the GPU (I don't know how to see how much GPU memory is being used on macOS)
(DIR) Post #AcJOApithXmnVoj4WO by mergesort@macaw.social
2023-11-29T21:50:57Z
0 likes, 0 repeats
@simon Got it, thanks a lot! I have an 24GB M2 Air that tends to struggle a bit with local models. I'm not surprised about that, but that's why I was curious if it was more powerful specs or just the models/llamafile that made this better.
(DIR) Post #AcJQ4Z72kJ2SAN9v0q by fpbhb@mastodon.social
2023-11-29T22:12:09Z
0 likes, 0 repeats
@simon ... and now I'm more amazed by goold old understandable tech and study Cosmopolitan instead of learning about LLMs ;-) Thanks for the pointer!
(DIR) Post #AcJTCJ1Ml1okREszqa by simon@fedi.simonwillison.net
2023-11-29T22:47:35Z
0 likes, 0 repeats
Added this last note to my post:"Stick that file on a USB stick and stash it in a drawer as insurance against a future apocalypse. You’ll never be without a language model ever again."
(DIR) Post #AcJTtDMFgl39KJ7Ehc by mattmay@mstdn.social
2023-11-29T22:55:05Z
0 likes, 0 repeats
@simon *if you remembered to install Xcode
(DIR) Post #AcJVeach6gLz2xxcGm by knur@tilde.zone
2023-11-29T23:14:47Z
0 likes, 0 repeats
@simon That's amazing... and it uses Justin Tunney's cosmopolitan???Everything project from her is both a piece of art and marvel of engineering.This is exciting stuff :)
(DIR) Post #AcJWARgBtVf2ZjmnQ0 by glyph@mastodon.social
2023-11-29T23:15:39Z
0 likes, 0 repeats
@mattmay @simon … and if you remember to take it out of the drawer before the NAND flash decays because it hasn't had a refresh …
(DIR) Post #AcJWASbyQfsjSwowU4 by simon@fedi.simonwillison.net
2023-11-29T23:20:46Z
0 likes, 0 repeats
@glyph @mattmay hah, yeah maybe burning it to a DVD would be better?What's the current best way to store 4GB offline so it will be reusable in 30 years time?
(DIR) Post #AcJWYKxmW6iShAPFx2 by SnoopJ@hachyderm.io
2023-11-29T23:21:41Z
0 likes, 0 repeats
@simon @glyph @mattmay placing my bet for "tape" right now
(DIR) Post #AcJWYNzfFZ0M5sJIno by SnoopJ@hachyderm.io
2023-11-29T23:22:40Z
0 likes, 0 repeats
@simon @glyph @mattmay a plain old spinning-platters hard drive might also be an option, I don't really recall how the two compare on this time-scale.
(DIR) Post #AcJWkZc3sLKJbb57yK by glyph@mastodon.social
2023-11-29T23:24:20Z
0 likes, 0 repeats
@simon @mattmay I think M-Disc is still probably your best bet, given its intentional shelf stability and resistance to magnets since every portal computing device now has a bunch of ridiculously powerful neodymium glued to it. As @SnoopJ suggests, tape is probably still more technically long-term durable at a physical level in a controlled environment, but finding a working tape reader in 20 years is going to be even harder than finding a working optical drive
(DIR) Post #AcJWxFGcpdCuGj66gS by lewiscowles1986@phpc.social
2023-11-29T23:27:59Z
0 likes, 0 repeats
@simon @glyph @mattmay Solid-State-StorageI'd be willing to bet that most thumbdrives if regularly charged will survive decades.
(DIR) Post #AcJXMb4EurM37XOPcu by saurabhs@mas.to
2023-11-29T23:30:09Z
0 likes, 0 repeats
@simon What % of your personal LLM use do you use on a hosted LLM like chatgpt vs running locally?
(DIR) Post #AcJXZn0dpMW9vG05aK by mattmay@mstdn.social
2023-11-29T23:32:45Z
0 likes, 0 repeats
@simon @glyph If I were worried about never having connectivity again (ftr, I am not), I would focus less on the durability of the medium, and more on how recent a snapshot I could throw in my bugout bag. 30 years from now, I don't want the 486DX of LLMs.In which case, I'd probably put a SFF PC on a UPS, with a USB stick in it, update the LLM every month, and write it out to USB every time the network or power went down.Maybe 2 USB drives. ;)
(DIR) Post #AcJYn2m6cfLWXN5MjQ by simon@fedi.simonwillison.net
2023-11-29T23:50:18Z
0 likes, 0 repeats
@saurabhs honestly 95% of my personal use is against hosted LLMs - almost entirely ChatGPT through their UI or gpt-3.5-turbo or gpt-4-turbo via their APIsI love tinkering with the local ones but for actual work I tend to stick with the best available hosted models
(DIR) Post #AcJdHS9JFJ5hjph7pI by pimont@mastodon.social
2023-11-30T00:40:24Z
0 likes, 0 repeats
@simon This is fantastic. Any advice for accessing the '/completion' API endpoint from a rest client or other web app?All I get is 'Connection was refused by the server' at the moment. 🤔
(DIR) Post #AcJh1akTuAIdanm1sO by rspfau@ecoevo.social
2023-11-30T01:22:17Z
0 likes, 0 repeats
@simon I would say the last sentence describing the plant is false. It's an ugly mess.
(DIR) Post #AcJhRRVgsNwWT8BLsG by dws@mastodon.cloud
2023-11-30T01:26:33Z
0 likes, 0 repeats
@simon About 1 token every 6 seconds on an ancient i3 NUC running Ubuntu 20.04. Still, directionally way cool!
(DIR) Post #AcJq88jfM3DuKGac6K by billseitz@toolsforthought.social
2023-11-30T03:04:19Z
0 likes, 0 repeats
@simon"can I eat this mushroom?""sure why not"Problem solved.
(DIR) Post #AcJqoL2ZzkjYeTYem0 by billseitz@toolsforthought.social
2023-11-30T03:11:38Z
0 likes, 0 repeats
@simon come the apocalypse how many bike-hours will it take to get an answer?http://webseitz.fluxent.com/wiki/UnCivilizationTechnologyhttps://www.npr.org/sections/13.7/2016/12/08/504790589/could-you-power-your-home-with-a-bike#:~:text=Nope.,by%20a%20100%2Dwatt%20lightbulb.
(DIR) Post #AcK5gaxgPqAl1f28iO by vick21@mastodon.social
2023-11-30T05:58:42Z
0 likes, 0 repeats
@simon @jcsteh Already on it! Runs blazingly fast on my Mac mini M2 pro. Wow! LEt’s see if it has an API.
(DIR) Post #AcKDc3ZpOfQRxZDqT2 by ftranschel@norden.social
2023-11-30T07:27:29Z
0 likes, 0 repeats
@simon I truly like your work, but I have a request: Please don't use the term *magic* in context of LLMs in particular and deep learning in general. It's boderline misleading and dangerous to imply that there is anything in this that would be hard to understand.
(DIR) Post #AcKGobBgYiV5XSbNtQ by olaf_brandt@openbiblio.social
2023-11-30T08:03:01Z
0 likes, 0 repeats
@simon @glyph @mattmay Storage Technology is just a smaller part of Digital Preservation.
(DIR) Post #AcKNovEEkIghNGtePg by anandphilipc@sigmoid.social
2023-11-30T09:21:30Z
0 likes, 0 repeats
@simon not able to get this to work on my ubuntu machine
(DIR) Post #AcKpRRV3uK4FEFMJYu by simon@fedi.simonwillison.net
2023-11-30T14:31:07Z
0 likes, 0 repeats
@ftranschel in this case the term "deep magic" is describing Cosmopolitan Libc, not LLMsI wrote about why I avoid the term magic with LLMs a while ago: https://simonwillison.net/2022/Oct/5/spell-casting/
(DIR) Post #AcKpfX7tmkr2jVF9nc by gabboman@app.wafrn.net
2023-11-30T10:08:01.000Z
0 likes, 0 repeats
@simon@fedi.simonwillison.net isnt llama the one that leaked from facebook?
(DIR) Post #AcKpfXy0f0XRL7cm1Y by simon@fedi.simonwillison.net
2023-11-30T14:33:47Z
0 likes, 0 repeats
@gabboman that story was overblown in my opinion: Facebook released LLaMA 1 to any researcher who filled in a form, then one of those researchers put it on a torrentThen in July Facebook released Llama 2 under a license that allowed almost everything, including commercial use - that's the model everyone is building on top of today
(DIR) Post #AcKr5rGPdbPMffLHKS by ftranschel@norden.social
2023-11-30T14:49:52Z
0 likes, 0 repeats
@simon Thank you very much.I think, my main gripe is (irrespective of whether it's specifically relating to LLMs or some other framework/concept) this: "I’ll continue to enthuse about spell-casting with fellow nerds who aren’t at risk of assuming these systems are incomprehensible magic"My experience is that this assumption is deeply flawed and there are more and more "code wizards" disguising as actual IT devs / Data Scientists / AI experts / whatever professionals.YMMV of course 🤝
(DIR) Post #AcMVq4M1UT3buqW4nI by almad@fosstodon.org
2023-12-01T10:00:59Z
0 likes, 0 repeats
@simon @glyph @mattmay I have bought dvd burner and m-disc for this reason, after being surprised how most of my $1 discs home burned on 2nd gen cd burners were still readable after 20y.
(DIR) Post #AdFPD2L6tUe1P0Lum8 by diti@dook.business
2023-12-27T21:35:52Z
0 likes, 0 repeats
@simon I just tried this out on a read-only system (NixOS). The llamafiles unfortunately use hardcoded paths (like /bin/mkdir – POSIX doesn’t specify those tools must live in /bin), so I have Bash errors. A brilliant idea, but “it doesn’t work on my machine”!