[HN Gopher] Many options for running Mistral models in your term...
___________________________________________________________________
Many options for running Mistral models in your terminal using LLM
Author : simonw
Score : 61 points
Date : 2023-12-18 18:27 UTC (4 hours ago)
(HTM) web link (simonwillison.net)
(TXT) w3m dump (simonwillison.net)
| bitdeep wrote:
| You missed ollama option
| M4v3R wrote:
| Which honestly is the easiest option of them all if you own an
| Apple Silicon based Mac. You just download the ollama and then
| run `ollama run mixtral` (or choose a quantization from their
| models page if you don't have enough ram to run the defalt q4
| model) and that's it.
| eurekin wrote:
| I tried an hour ago and had a can't load model error.
| Everything up to date. Is there any special step?
| mmcclure wrote:
| Tried `ollama pull mixtral` just now and it seems to be
| working, albeit pretty slowly.
| Casteil wrote:
| How much RAM do you have? Mixtral is a beast and the non
| quantized model wants 40GB+ of memory.
| simonw wrote:
| I was hoping I could run my LLM CLI tool against Ollama via
| their localhost API, but it looks like they don't offer an
| OpenAI-compatible endpoint yet.
|
| If they add that it will work out of the box:
| https://llm.datasette.io/en/stable/other-models.html#openai-...
|
| Otherwise someone would need to write a plugin for it, which
| would probably be pretty simple - I imagine it would look a bit
| like the llm-mistral plugin but adapted for the Ollama API
| design: https://github.com/simonw/llm-
| mistral/blob/main/llm_mistral....
| throwup238 wrote:
| Mixtral-8x7B-Instruct is also available in a llamafile now:
| https://github.com/Mozilla-Ocho/llamafile#other-example-llam...
| simonw wrote:
| Just added that option - you can talk to it from my LLM CLI
| tool using its OpenAI-compatible API endpoint:
| https://simonwillison.net/2023/Dec/18/mistral/#llamafile-ope...
| MilaM wrote:
| This is a bit off-topic and probably a noob question. But maybe
| someone with experience could help me. I'm about to buy a M2 or
| M3 Mac to replace my Intel Mac, and would like to play around
| with locally run LLMs in the future. How much RAM do I have to
| buy in order not to run into bottlenecks? I'm aware that you
| would probably want a beefier GPU for better performance. But as
| I said, I'm not worried about speed so much.
| azeirah wrote:
| If you want to run decently heavy models, I'd recommend getting
| at a minimum getting 48GB. This allows you to run 34b llama
| models with ease, 70b models quantized, mixtral without
| problems.
|
| If you want to run most models, get 64GB. This just gives you
| some more room to work with.
|
| If you want to run anything, get 128GB or more. Unquantized
| 70b? Check. Goliath 120b? Check.
|
| Note that high end consumer gpus end at 24GB VRAM. I have one
| 7900xtx for running llms, and the best it can reliably run is
| 4-bit quantized 34b models, anything larger is partially in
| regular ram.
| MilaM wrote:
| Thank you for this detailed response. I'm not sure if it was
| clear, but I was going to use just the Apple Silicon CPU/GPU,
| not an external one from Nvidia.
|
| Is there anything useful you can do with 24 or 32GB of RAM
| with llms? Regular M2 Mac minis can only be ordered with up
| to 24GB of RAM. The Pro Mac mini M2 is upgradable to 32GB
| RAM.
| simonw wrote:
| 32GB will run a good quantized Mixtral, though I can't
| confidently explain how much of a quality difference there
| is from unquantized.
| cschneid wrote:
| Data Point: I am currently having issues getting Mixtral
| Q4_K_M running in LMStudio on my 32gb M1 Max. I'm trying
| Q3 to see if it fits.
|
| I can have it run in on 'cpu' which is very slow, but
| offloading to the GPU runs out of memory.
| azeirah wrote:
| I was talking about m2 Macs. Just comparing that the best
| you can do with a gpu is 24GB, Macs go far beyond because
| of their integrated memory.
| MilaM wrote:
| Ok, sorry. I did not understand that you just mentioned
| that to give more context. Totally makes sense.
| robterrell wrote:
| I've been unable to get Mixtral
| (mixtral-8x7b-instruct-v0.1.Q6_K.gguf) to run well on my M2
| MacBook Air (24 GB). It's super slow and eventually freezes
| after about 12-15 tokens of a response. You should look at
| M3 options with more RAM -- 64 GB or even the weird-
| sounding 96 GB might be a good choice.
| itissid wrote:
| https://www.reddit.com/r/LocalLLaMA/comments/17kcgjv/how_
| doe... reddit thread talks about some of the pros and
| cons of a m3 max with 128 Gb costing ~5-6K
| simonw wrote:
| Get as much RAM as possible.
|
| 16GB is not enough.
|
| 32GB is enough to run quantized Mixtral, which is the current
| best openly licensed model.
|
| ... but who knows what will emerge in the next 12 months?
|
| I have 64GB and I'm regretting not shelling out for more.
|
| Frustratingly you still have WAY more options for running
| interesting models on a Linux or Windows NVIDIA device, but I
| like Mac for a bunch of other reasons.
| eurekin wrote:
| Why not both? :)
|
| Bait aside, I'd love to read about how are you using those
| models. I'm mostly interested in code comprehension and
| meeting summarisation.
| simonw wrote:
| I still mostly use GPT-4 to get actual work done, because
| until a Mixtral came along it felt like the local models
| were no competition.
|
| I'm going to bump up my usage of Mixtral a bit now to see
| how it feels for that kind of stuff.
| eurekin wrote:
| Ah, same here. Have been itching to make some use of
| those local llms for some time, always ending up in
| chatgpt as well.
|
| Although for those napkin like ideas gpt4 (including the
| turbo variant) get costly quickly.
| nextworddev wrote:
| Just got 64gb as well but about to refund it for a bigger one
| after reading your comment :)
| Casteil wrote:
| I run 7b/13b models pretty gracefully on a 16GB M1 Pro, but
| it does leave me wanting a little more headroom for other
| things like Docker and browser eating multiple gigabytes of
| ram themselves.
|
| Maybe keep an eye out for M1 / M2 deals with high ram config?
| I've seen 64GB MBPs lately for <$2300 (slickdeals.net)
| MilaM wrote:
| Thank you (and all the adjacent comments). I don't really
| need so much RAM for my regular work. Running the llms
| locally would only be for fun and experiments.
|
| I think 32GB might be the best middle ground for my needs and
| budget constraints.
|
| It's really a pity that you can't extend RAM in most Apple
| Silicon Macs and have to decide carefully upfront.
| amrrs wrote:
| I ended up ordering a 36GB M3 for similar reason.
|
| I currently run Mistral and a few mistral derivatives using
| Ollama with decent inference speed on a 2019 Intel Mac
| 32GB. So I assumed the new one with 32ish should do a
| better job.
|
| I've tried vision model Llava as well, a bit more latency
| but works fine.
|
| With Apple's own Mlx things might improve .
| loudmax wrote:
| TheBloke's GGUF model card tells you how much RAM you'll want
| to run the various versions of the model:
| https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF#provi...
|
| In the article, Simon mentions the Q6_K.gguf model, which is
| about 40GB. A Mac Studio can handle this, but any of these
| models are going to be a tight fit or impossible on a Mac
| laptop without swapping to disk. Maybe NVME is fast enough that
| swapping isn't too terrible.
|
| In my experience, the Mixtral models work pretty well on
| llama.cpp on my Linux workstation with a 10GB GPU, and
| offloading the rest to CPU.
|
| It is impressive how fast the smaller models are improving.
| Still, a safe rule of thumb is the more RAM the better.
|
| Also, really question how much you need to run these models
| locally. If you just want to play around with these models,
| it's probably far more cost effective to rent something in the
| cloud.
| williamstein wrote:
| I tried Mixtral via ollama on my Apple M1 Max with 32GB of
| RAM, and was a total nonstarter. I ended up having to
| powercycle my machine. I then just used two L4 GPU's on
| Google Cloud (so 48GB of GPU RAM, see [1]) and it was very
| smooth and fast there.
|
| [1] https://github.com/sagemathinc/cocalc-
| howto/blob/main/ollama...
| avereveard wrote:
| long story short, for a machine that has about 5+ years of
| shelf life it isn't a bad value prop to go and buy the absolute
| top end, especially if you expect some roi out of it, just be
| aware that if you don't want the top end for llm the m2
| currently provides more value because m2 has memory bandwith
| than the m3 at the middle range.
|
| memory bandwidth is the key to model speed, and memory size is
| what enable you to use larger model (quantization let you push
| thing further, to a point) so one thing to note is that on the
| M3 pro/max only the top end model gets the full bandwidh, while
| the m1/m2 pro enjoy full bandwidth from a smaller memory size.
| this may be important if you value speed above model size or
| vice versa. M2 Pro, M2 Max get approximately 200 GB/s and 400
| GB/s, but things are more complicated for m3: M3 Pro gets
| 150mb/s, and M3 max gets 300mb/s at 36gb and 400mb/s at 48gb
|
| few more things to note:
|
| it's absolutely fine to go and play around with llm but even
| with a llm monster machine there's nothing wrong in starting
| with smaller models and learning how to squeeze the maximum
| amount of work out of them. the learning do transfer to larger
| model. this may or may not be important if at some point you'll
| want to monetize or deploy to production what you learned.
| while the mac itself is a good investment for personal use,
| once you move to servers, cost skyrockets with model size,
| because of supply constraints on 40gb+ memory gpus. if you are
| dependent to a 70b parameter model, you'll have a hard time to
| make a cost effective solution. if it's stricly to playing
| around, you can disregard this concern
|
| even if you're playing around, a 70b is going to run at 7
| tokens / second, which is fine for a local chat, but if you are
| writing a program and need inference in bulk, it's fairly slow.
|
| another thing of note is that while the field is still
| undecided on which size and architecture is good enough, the
| moltitude of small fish experimenting with tuning and mix of
| instructions are largely experimenting on smaller models.
| currently my favorite is openhermes-2.5-mistral-7b-16k, but
| it's not an indication that mistrals are strictly better than
| llama2, more an indication that experimenting with 7b is more
| cost effective for third parties without access to gpu than
| experimenting with 13b, and so you'll find 13b model kinda
| stagnating, with many of them trained in a period where people
| didn't really know the best parameters for finetuning and are
| so to say a bit behind the curve. a few tuners are working at
| 70b models, but these seems to be pivoting to mixtrals and the
| likes, which will cause a similar stagnation on the top end,
| that is, until llama3 or the next mixtral size drops, then, who
| knows
| whiplash451 wrote:
| Rule of thumb: when buying a new Mac, always max RAM.
| itissid wrote:
| https://www.reddit.com/r/LocalLLaMA/comments/17kcgjv/how_doe...
| reddit thread has some ideas.
|
| What I can draw from reading of that thread is that you can buy
| a Desktop Rig with 200GB memory bandwidth (comparable to m3 pro
| and max) and a lot of expansion capability (256GB RAM). You
| should find out if that's still good enough for your local use
| case for token per second or training.
|
| Then just use SSH/XTerm(and possibly ngrok) to login with good
| speed from anywhere into your rig with a light M2 ?
| mrbonner wrote:
| Any option to use a SaaS UI to use those OS models? I'm ok with
| paying the API access for Anya ale for example and use the ApI
| for the UI.
| ntonozzi wrote:
| I've been using baseten (https://baseten.co) and it's been fun
| and has reasonable prices. Sometimes you can run some of these
| models from the hugging face model page, but it's hit or miss.
| Der_Einzige wrote:
| All this needless difficulty when you can just do the right thing
| and use oogabooga.
|
| https://github.com/oobabooga/text-generation-webui
| simonw wrote:
| Looks like that provides an OpenAI-compatible API endpoint,
| which means you can use my LLM command-line utility against
| with the same pattern as for Llamafile:
| https://simonwillison.net/2023/Dec/18/mistral/#llamafile-ope...
| brcmthrowaway wrote:
| This may be a dumb question but how do you update an LLM with new
| information after its knowledge cutoff?
| eurekin wrote:
| - in context learning (simply put it into context)
|
| - finetuning, as in restarting the models checkpoint and
| relearning it on the previous + new data
|
| - adding extra neurons (e.g. LoRA adapters) at certain places
| and restarting learning
|
| Oh, in classic machine learning there's also the
| "bagging/boosting classifiers" option, but I have no knowledge
| if that can be applied to a ANN.
| simonw wrote:
| Have you seen LoRA adapters actually work for this kind of
| thing?
|
| The leaked Google "We have no moat" memo was very excited
| about LoRA style techniques, but it's not clear to me that
| it's been proven as a technique yet.
| eurekin wrote:
| No, I have tried about 3 times and all failed. Altough it
| might be that I'm simply doing it wrong.
|
| There are people (can point at a discord server) claiming
| it works for them and that they even sell finetuned models
| to business clients.
|
| EDIT: I found one of the articles I tried to follow:
| https://www.mlexpert.io/prompt-engineering/chatbot-with-
| loca...
|
| EDIT2: Ignore above. This seemed much more promising:
| https://www.youtube.com/watch?v=pnwVz64jNvw . Author
| provides consulting services and seemed very nice and
| approachable
| simonw wrote:
| Generally, you train it again entirely from scratch.
|
| It's possible to introduce new information by fine-tuning a new
| model on top of the existing model, but it's debatable how
| effective that is for introducing new information - most fine-
| tuning success stories I've seen focus on teaching a model how
| to perform new kinds of task as opposed to teaching it new
| "facts".
|
| If you want a model to have access to updated information, the
| best way to do that is still via Retrieval Augmented
| Generation. That's the fancy name for the trick where you give
| the model the ability to run searches for information relevant
| to the user's questions and then invisibly paste that content
| into the prompt - effectively what Bing, Bard and ChatGPT do
| when they run searches against their attached search engines.
| MacsHeadroom wrote:
| The easiest way is to let it do web searches with a tool
| former/plugin framework.
|
| For specifically knowledge you want it to be able to recall
| (like knowledge base articles or blog posts) vector database
| embeddings are best.
|
| For knowledge you want it to operationalize, like being able to
| program in a new language the last resort is finetuning but
| this is not easy, requires massive amounts of high quality
| data, and is not generally effective for things which do not
| have a large amount of data to fine tune on (tens of thousands
| of pages worth of content).
| loudmax wrote:
| I don't think that's a dumb question at all. It depends on your
| objectives and how much resources you're willing to spend.
|
| These open weights models can be retrained. Start with a
| foundational model like Llama2 or something and expose it to
| more recent training data that includes whatever updated
| information you want it to have access to. This is relatively
| expensive, but allows for big changes to the model.
|
| If you have some relatively small subset of new information you
| want to bring in, you could build a Lora. Then either run your
| model with the Lora, or fold the Lora into your base model.
| This is relatively cheap, but fairly narrow in terms of your
| updates.
|
| In the long run, it might be that Retrieval Augmented
| Generation (RAG) is the way to go. Here, your embeddings go
| into a vector database, and the model reads from there. Then
| you just need to update the database for the model to have
| access to new information.
|
| This LLM stuff is new enough that anything like best practices
| are still being worked out. The optimal way to bring in new
| information could be a variant of one of the methods I
| mentioned above, or some combination of all three, or something
| else altogether.
| pkaler wrote:
| The term that you're looking for is Retrieval Augmented
| Generation (RAG).
|
| https://docs.aws.amazon.com/sagemaker/latest/dg/jumpstart-fo...
| syntaxing wrote:
| Anybody have a Mistral invitation they can spare? I'm super
| curious how the medium performs compared to GPT3.5/4
| simonw wrote:
| Are you on their waitlist? I got through that in less than 48
| hours, so it might be worth a shot.
| syntaxing wrote:
| I am, I signed up late though and got an email about it last
| night " Access to our API is currently invitation-only, but
| we'll let you know when you can subscribe to get access to
| our best models."
| kristianpaul wrote:
| I run in Linux system it via cli using Ollama, very easy to setup
|
| https://ollama.ai/library/mistral
|
| curl https://ollama.ai/install.sh | sh
|
| ollama run mistral:text
___________________________________________________________________
(page generated 2023-12-18 23:02 UTC)