[HN Gopher] Llamafile is the new best way to run a LLM on your o...
___________________________________________________________________
Llamafile is the new best way to run a LLM on your own computer
Author : djha-skin
Score : 139 points
Date : 2023-12-01 17:36 UTC (5 hours ago)
(HTM) web link (simonwillison.net)
(TXT) w3m dump (simonwillison.net)
| manojlds wrote:
| Previous: https://news.ycombinator.com/item?id=38464057
| dang wrote:
| Thanks! Macroexpanded:
|
| _Llamafile lets you distribute and run LLMs with a single
| file_ - https://news.ycombinator.com/item?id=38464057 - Nov
| 2023 (273 comments)
| behnamoh wrote:
| dupe
| bilsbie wrote:
| Is there a good machine to buy in the $500-1000 range that is
| able to run some of this stuff?
| _neil wrote:
| Maybe an M2 Mac mini?
| robterrell wrote:
| Don't skimp on RAM.
| superkuh wrote:
| Pretty much any modern machine will do you fine for ~5 tokens/s
| on a 7B (small end) model like Mistral-7B or llama2-chat-7B (or
| any of their respective fine-tunes). The computer you already
| have can probably do this.
| StillBored wrote:
| It can run on CPU cores if you have enough RAM for the model.
| It feels like you have time warped back to the early 1990's and
| are talking to someone on a BBS (AKA the words appear slowly),
| but it is entirely functional if you have 32+GB of RAM.
| nielsole wrote:
| For reference: I run 7B 5bit quantized models on a ryzen 7
| 5700G with 64Gb at 8 tokens/second CPU only. It's not close
| to what you can get with a high end graphics card but for
| every day use it is alright and has headroom for bigger
| models. Upgrading CPU, RAM and Mainboard in a 10+year old PC
| cost me just 400EUR
| tarruda wrote:
| I haven't tested, but I suspect you might be able to get good
| CPU inference on this mini pc:
| https://www.aliexpress.com/item/1005005825981362.html. It costs
| ~$800 in the maxed configuration with 64GB RAM clocked at
| 5600Mhz
| mratsim wrote:
| Hopefully, AMD wakes up and allows ROCm / HIP on their 7040
| series, that would be a killer feature.
|
| And AFAIK the Ryzen pro do have an "AI coprocessor" but
| unsure of the API to use them.
| swuecho wrote:
| mac mini 8G runs at 10tokens/sec
| lxe wrote:
| Keep in mind that gguf/llama.cpp, although highly performant and
| portable, is not the best performing way to launch certain models
| if you have a GPU. (even though llama.cpp does support GPU
| acceleration)
|
| ExLLAmA v2 + elx2 quantization, and maybe tensorrt-llm might be
| the contender for the top performer
| ComputerGuru wrote:
| How much more work is it to get those up and running?
| idonotknowwhy wrote:
| Almost none of you already have python. Download exl2, exui
| from github and run a few terminal commands. This let's me
| run the 120b param models, which won't fit in vram if I use
| llamacpp
| superkuh wrote:
| It's not the best way. It's a really cool and technically
| interesting way. But embedding the model with the executable is
| terrible for anything beyond a demo. The best way would be just
| running llama.cpp (what llamafile uses) and loading external
| models. I get that compiling is too difficult for some people and
| llamafiles are great for them. Maybe even the best for them.
|
| But it's far from best for people that actually want to explore
| LLM and play.
| breckenedge wrote:
| Per the article:
|
| > You can also download a much smaller llamafile binary from
| their releases, which can then execute any model that has been
| compiled to GGUF format
| rodrigobellusci wrote:
| From what I've read llamafile can also load other models if you
| pass a flag with a path to them. There's also LM Studio for
| anyone who'd like to play with a ChatGPT-like GUI.
| brandall10 wrote:
| I'd argue LM Studio is the best way by far.
|
| - Download and launch the app
|
| - Use their interface to huggingface models, download the one
| you want
|
| - Go to the chat window and select load
|
| - Chat away w/ a ChatGPT like interface that includes markdown
| processing and easy copying of code blocks, etc
|
| Outside of the time to download a model, it's about 30 seconds
| of work to get up and running
| leonidbelyaev wrote:
| Doesn't work on my NixOS workstation.
| SkyMarshal wrote:
| Why not, what's the error, and how did you set it up? Any
| chance you tried it in a nix shell or nix vm or other isolated
| environment?
| psanford wrote:
| It fails because it hardcodes the path to certain binaries
| instead of looking in PATH: `line 60: /bin/mkdir: No such
| file or directory`
| SkyMarshal wrote:
| Oh lol, I've never even heard of /bin/mkdir on regular FHS
| linuxes. Is that actually a thing?
| FragenAntworten wrote:
| See this comment from the other discussion:
| https://news.ycombinator.com/item?id=38469749
| rasengan wrote:
| This is probably true for certain scenarios, but it's absolutely
| not true for all scenarios.
| Akashic101 wrote:
| I am currently planning my own small LLM trained on documents we
| use internally for work. Does anyone have any tips and tricks on
| how to make this work the best? Could a project like Llamafile
| help me with this, even if it is just for testing?
| bonniemuffin wrote:
| How did you choose between training a model from scratch vs
| using retrieval augmented generation with an existing off-the-
| shelf model? From what I've observed, RAG + off-the-shelf model
| seems to be the more common approach for use cases like "create
| LLM that answers questions about my company's internal
| documentation", particularly because the iteration/improvement
| cycle is much shorter-- it's much easier to iterate on
| RAG/prompts vs. training a whole new model to improve it. (If
| the answer is "I just wanted to try training a whole new llm",
| I won't fault you for that! :) )
| youniverse wrote:
| Anyone know if this (or any LLM) is worth running locally on a
| Macbook M2 Air?
|
| I'll give it a shot over the weekend but if anyone knows I'm
| curious!
| buffington wrote:
| Without knowing what you consider to be of worth, that's
| difficult to answer.
|
| The good news is that the article describes in detail how to
| determine for yourself if it's worth it. The author uses an M2
| as well, so you can at least know it'll likely work.
| MPSimmons wrote:
| It depends on what you want them to do and what you want to get
| out of them. A lot of people use less machine than a M2 Air for
| different things.
| brandall10 wrote:
| If you have 8GB you can play around with heavily quantized 7B
| models, or up to moderately quantized 13GB models w/ 16GB.
|
| Something like a Mistral-7B Dolphin finetune actually is
| surprisingly useful, like GPT-3.5 in some respects. I imagine
| it would render ~10 tok/sec on an M2 for at least short bursts
| until throttling sets in.
|
| As I mentioned in another post, try this out with LM Studio.
| Super simple GUI that even a non-tech person could probably
| figure out for finding/downloading/loading models w/ a ChatGPT
| like interface for chatting.
|
| https://lmstudio.ai/
| dragonwriter wrote:
| Best in what sense? Seems by description a whole lot less
| convenient (and more limited) if you want to use more than one
| model (and especially more than just models available as GGUFs)
| than oobabooga/text-generation-webui or other similar tools that
| do things like bundle multiple backends for different LLM
| architectures, support downloading models from huggingface,
| present a common web UI for LLM configuration, managing prompts,
| and actually doing inference (in chat, notebook, and other
| styles), and also supports presenting an OpenAI-compatible API
| endpoint backed with a local LLM to support other frontends.
| rismay wrote:
| Will this approach work with image generation models?
___________________________________________________________________
(page generated 2023-12-01 23:01 UTC)