[HN Gopher] MLC-LLM: GPT/Llama on consumer-class GPUs and phones
___________________________________________________________________
MLC-LLM: GPT/Llama on consumer-class GPUs and phones
Author : junrushao1994
Score : 251 points
Date : 2023-04-30 15:30 UTC (7 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| TheObviousOne wrote:
| is there any integration with it to langchain?
|
| +
|
| Is there any optimization for LLM to run on RTX cards? 40XX,30XX
| I found out tha LLAMA.CPP is nice but I want to take advantage of
| my graphic cards also, and didn't found any documentations...
| juliangoldsmith wrote:
| rllama has an OpenCL version, though I wasn't able to test it.
| homarp wrote:
| "Our primary workflow is based on Apache TVM Unity, an exciting
| ongoing development in the Apache TVM Community."
| TheObviousOne wrote:
| Is there a way to make it answers longer answers?
| syntaxing wrote:
| What sort of performance would you expect on a P40 with either 4
| bit or 8 bit GPTQ 13B? My biggest issue with Triton is the lack
| of support for Pascal and older GPUs. With CUDA, I only get about
| 1-3 tokens per second.
|
| Are these the only supported models as of now?
| https://github.com/mlc-ai/mlc-llm/blob/d3e7f16c54238b7da5e78...
| eurekin wrote:
| Not wanting to derail the thread, but could be the best place to
| ask this:
|
| What are you using local LLM's for?
|
| So far, I've been only able to come up with:
|
| - Aid in coding (which always ends up in chatGPT)
|
| - Summarizing short articles
|
| - whisper-ai + langchain + ffmpeg allows for some great video
| summarization (especially with non-english LORA's for us non-
| natives)
|
| - generating stable diffusion prompts
| flatiron wrote:
| ive been playing with it locally in the hopes of a model that
| allows for commercial use. at my job if i had a model I could
| run in the cloud and just wrap a REST service around I could
| think of a ton of ways to use it both internally and
| externally.
| eurekin wrote:
| Thanks! If ChatGPT can be used commercially and it
| successfully passed SOC3 cert, wouldn't you still want to use
| those non - chatgpt models?
|
| Also, you hint at those many ideas, could you elaborate on
| that a bit? I'll be playing with LLMs in near future, might
| as well do something useful with them
| flatiron wrote:
| my concern with using chatgpt is PII. If I host the LLM and
| set it up that it doesn't record any of the interactions
| besides some weird meta data and sign that in a contract i
| bet a bunch of my clients would like to use my LLM.
| especially if I can train it on internal documentation that
| they can't/won't send to a big third party like chatgpt. im
| one throat to choke and i already have their PII so i think
| its a good fit.
|
| without getting into too much detail my job supports
| business to people interactions. my use case is training
| the LLM to assist the business agents. if it can give real
| time information that's helpful to the agent while causally
| listening to the conversation thats a pretty big game
| changer. also i want to use it for staffing decisions since
| it can view historic data and make recommendations for the
| future.
| UncleEntity wrote:
| How big of a model do you suppose it would take to come
| up with a Punctuation-as-a-Service cloud-based solution?
|
| Asking for a friend...
| int_19h wrote:
| The installation process includes downloading precompiled
| binaries from this repo:
|
| https://github.com/mlc-ai/binary-mlc-llm-libs
|
| Is the code from which these are built available somewhere? How
| does one go about building one for their own model?
| raverbashing wrote:
| I wouldn't be surprised if the iPhone (or other phones) come with
| a LLM pre-built as a Siri/Hey Google replacement
|
| (on the other hand I wouldn't be surprised if they didn't come
| with it neither, due to the difficulties of it)
| seydor wrote:
| Would consume too much battery. Will probably remain in the
| cloud
| Hippocrates wrote:
| I'm 100% sure this will happen, and soon.
| aezart wrote:
| I can already run a language model on my GPU using minillm or
| text-generation-webui, and on my CPU using llama.cpp. What makes
| MLC-LLM better?
| simonw wrote:
| Have you got those to work on an iPhone?
| sroussey wrote:
| Have it working on mine!
| sroussey wrote:
| That works on a Mac?
| junrushao1994 wrote:
| You no longer need a powerful latest-gen GPU to run SOTA
| models, plus going through complicated setups. MLC-LLM makes it
| possible to use GPUs from any vendors, including
| AMD/Apple/NV/Intel, to run LLMs at reasonable speed, at any
| platform (win/linux/macos), even a steam deck :-)
|
| The way we make this happen is via compiling to native graphics
| APIs, particularly Vulkan/Metal/CUDA, making it possible to run
| with good performance.
| tyfon wrote:
| llama.cpp is not using the GPU, it runs fine on the CPU (if
| fast enough)
|
| I've scoured the web page for ram requirements for the
| various models but I can't see anything, will it be able to
| run let's say the 30B open assistant llama or 65B raw llama
| model on a consumer gpu (let's say 3060 with 12gb vram) using
| this?
|
| Not trying to take anything away, but the readme etc is very
| lacking in actual technical details I feel without reading
| through the code or actually testing it.
| junrushao1994 wrote:
| Thanks for the feedback! This is definitely something we
| need to do. To share some data, currently the default model
| is Vicuna-7b, aggressively quantized to 2.9G.
|
| We are expanding the coverage to more models, particularly,
| Dolly and StableLM are just around the corner, needing some
| clean up work.
|
| As a fresh new project, right now we are starting to
| collect data points of which GPU models are supported well
| and fixing issues being reported. Please don't hesitate to
| report in our github issue!
| tyfon wrote:
| I see, the 2.9 GB requirements seems to imply a 3 bit
| weights?
|
| In any case I am happy to see these projects taking form.
| Perhaps one can eventually make the level of quantization
| dynamic based on the available vram etc :)
|
| I will definitively play around with it (on linux though,
| not a phone!)
| int_19h wrote:
| When people tried 3-bit quantization for 7B models
| before, it did not exactly go well in terms of
| detrimental side effects. Are you using some new
| quantization techniques that mitigate that?
| azeirah wrote:
| Llama.cpp recently added partial GPU acceleration. Model
| dequantization as well as some BLAS operations have been
| moved to GPU.
|
| It runs a lot faster if you compile with cuBLAS (nvidia) or
| clblast (other). GPU vram doesn't matter much since it
| doesn't offload the model to vram.
| eulers_secret wrote:
| The local llama subreddit wiki has good info about RAM
| requirements:
| https://www.reddit.com/r/LocalLLaMA/wiki/models/
| junrushao1994 wrote:
| This is our latest project on making LLMs accessible to everyone.
| With this project, users no longer need to spend a fortune on
| huge VRAM, top-of-the-line GPUs, or powerful workstations to run
| LLMs at an acceptable speed. A consumer-grade GPU from years ago
| should suffice, or even a phone with enough memory.
|
| Our approach leverages TVM Unity, a machine learning compiler
| that supports compiling GPT/Llama models to a diverse set of
| targets, including Metal, Vulkan, CUDA, ROCm, and more.
| Particularly, we've found Vulkan great because it's readily
| supported by a wide range of GPUs, including AMD and Intel's.
|
| BTW, an interesting data point from Reddit that it also works on
| steam deck:
| https://www.reddit.com/r/LocalLLaMA/comments/132igcy/comment....
| cryptoboid wrote:
| No Android? :(
| junrushao1994 wrote:
| upcoming
| akrymski wrote:
| Nobody wants to run Google on their PCs, why should LLMs be
| different? I'd expect GPT models to be updated regularly fairly
| soon, and in much the same way that I wouldn't want to host a
| personal out-dated web index + search engine, LLMs seem a perfect
| fit for server-side services given their requirements. Barely
| anyone even hosts their blogs or mail. What's the excitement
| about getting it almost running on a phone about?
| leodriesch wrote:
| I think the privacy aspect is also interesting, think about
| journal or second-brain type applications. These could benefit
| a lot from language model use, but you don't want these types
| of information sent to a cloud provider in unencrypted fashion.
| whitepaint wrote:
| The easier to run it the more competition will emerge.
| int_19h wrote:
| LLMs are much more than a Google Search replacement, and many
| interesting use cases require them to have access to private
| data.
| chpatrick wrote:
| Because it's a world-changing technology and right now a
| handful of companies control it.
| charcircuit wrote:
| >why should LLMs be different?
|
| Because LLMs are expensive to host. It's more of a case that no
| one wants to run these on their PCs and it's the cheapest if it
| ends up running on client PCs instead of your own PCs. Not all
| use cases of LLMs need a super powerful model that is always up
| to date.
| mcemilg wrote:
| I think it is important to keep your that in device. I would
| prefer the chatgpt to work on my pc instead work on a giant
| companies servers. I am doing very sensitive conversations with
| it. There is no guarantee of this data to be kept secret by
| Open AI. Also corporation would want to keep their data in
| house. Especially it can be a huge problem if your developer
| can unleashed the private document with the internet.
| mrighele wrote:
| > What's the excitement about getting it almost running on a
| phone about?
|
| You get to decide what is appropriate or not.
|
| It works offline.
|
| It can be used to by applications without the having to use an
| external service.
|
| This can be important for a number of applications (I am
| thinking about open source games and the modding community
| right now, but it is just an example)
| whistle650 wrote:
| This is a great project thank you. I've installed the TestFlight
| app. FYI, right now it's saying in response to "Who was the
| president in 1973" that it was "Gerald Ford" which is wrong.
| matthewdgreen wrote:
| It's fun to ask it questions about famous computer scientists
| like Ron Rivest. Who is apparently a professor at Harvey Mudd
| College.
| eurekin wrote:
| Even non-quantized large LLMS (70b) have a lot of difficulties
| remembering facts. Chatgpt, being much larger, hallucinates a
| ton. It seems that it's not the best use case for them right
| now. Being a fact base that is
| QuadrupleA wrote:
| Does this support int4 tensor core operations on the Nvidia
| Turing & Ampere architectures? From what I've researched this
| would be a huge untapped speedup and memory saver for inference,
| but it's mostly undocumented, unsupported by pytorch etc. It'd
| basically be like llama.cpp but with a 10x or more GPU speedup,
| if someone was willing to dig in and write the CUDA logic for it.
| Given how fast llama.cpp already is on the CPU, this would be
| impressive to see.
|
| I've been tempted to try it myself, but then the thought of
| faster LLaMA / Alpaca / Vicuna 7B when I already have cheap gpt-
| turbo-3.5 access (a better model in most ways) was never
| compelling enough to justify wading into weird semi-documented
| hardware.
| qeternity wrote:
| The 7b model is ca. 6gb of VRAM so yes, it is already 4bit
| quantized.
|
| There are already efforts underway with GPTQ libraries but I
| have found they incur a substantial performance penalty, with
| the benefit of consuming much lower VRAM.
|
| EDIT: I had a look at the repo, it appears the Vicuna model is
| using 3bit quantization.
| junrushao1994 wrote:
| TVM Unity has a CUDA backend, and TensorCore MMA instructions
| are supported, so it wouldn't be hard to turn this option on.
|
| It's on our plan, but we haven't looked to enable them by
| default in the first place, mainly because we wanted to
| demonstrate it running on all GPUs including old models that
| don't come with TensorCore at all.
| jrm4 wrote:
| Anyone trying this out now? I mostly blindly copied and pasted
| the instructions and it's not slow -- but I'm getting pure Zalgo
| here...(RX 570 fwiw?)
| eurekin wrote:
| This field is in crazy progress mode now. Not long ago it was
| rent cuda gpu only. Now this... AMD could easily chip away some
| part of the market, if they released a > 24 GB vram gpu now.
| valine wrote:
| I hope they do, and I hope it forces Nvidia to release their
| own 48GB+ consumer card. 80GB is on my long term wish list as
| it would allow running a 65B model 8bit quantized. I don't see
| local models exceeding ChatGPT performance until we get to a
| point where folks can run 65B parameter models.
| yieldcrv wrote:
| yeah agreed, its sad to me that we're 2 months after llama
| and "nobody" is seemingly doing any advances of fine tuning
| on models with more than 7B or 13B parameters.
|
| I have 64gb RAM (not gpu just normal), I'd like to see proof
| of concepts that the bigger models can be fine tuned and have
| far more accepted results, or to know if we're completely
| going the wrong direction with this
| nullsense wrote:
| I find, since 30B models are actually quite usable locally
| if you have good hardware, that I really want something
| like a Vicuna 30B. That would be amazing. I can only run
| 65B locally at a speed of 1 token per second, which is too
| slow to be usable unfortunately.
| UncleEntity wrote:
| Probably because it's extremely affordable to train the
| smaller models.
|
| If I had the gumption (and a data set) I could afford to
| spend a few hundred bucks to fine tune a model for shits
| and giggles and I'm just a Random Internet Dude.
|
| I'm all for it, Any Day Now(tm) I have this idea I want to
| try and having these people do all this optimization work
| will probably make it affordable to attempt given I don't
| actually know what I'm doing so there will be a whole lot
| of "yeah, that doesn't work" going on.
| [deleted]
| amelius wrote:
| Wouldn't it be better to get a GPU that shares its memory
| with the CPU, like in the new macbooks?
| valine wrote:
| Sure, but other than Apple no one has the tight integration
| needed to make that happen. If Apple leans into their
| CoreML stuff they have a real opportunity to steal the
| market from Nvidia.
|
| Long term maybe Nvidia is able to release release some
| integrated ARM chip, but I'm not holding my breath.
| dhruvdh wrote:
| How do you imagine consoles work? And what about Apple's
| tight integration has actually made itself useful? The
| developer experience is terrible.
| valine wrote:
| It's useful because you can buy a Macbook with 64GB of
| RAM, and then use that RAM as VRAM to run your LLM.
| verdverm wrote:
| Nvidia sells data center products, Apple is not going to
| steal this market, they aren't even in it. The gaming
| market is dominated by Windows, I don't see this moving
| meaningfully just because apple has some marginally
| better hardware at this point in time.
|
| The Risc V area, now there we can talk about disruption
| long term
| amelius wrote:
| NVidia Jetson is a line of products with tight
| integration between CPU and GPU. As just one example.
| int_19h wrote:
| Unless you're doing training, is there much point in 8-bit
| for this model size? My understanding is that the larger the
| model, the less affected it is by quantization; for 65b,
| 4-bit gives you ~2% perplexity penalty over 8-bit.
| hawski wrote:
| Question from a noob: how good would it be to run those on a
| computer with AMD APU (for example Ryzen 9 7940HS) with 128GB
| RAM and setting aside 64GB for iGPU?
| crowwork wrote:
| You can try out the demo and benchmark yourself
| eurekin wrote:
| Another noob here. If I had to guess, it's because current
| models are mostly memory bound. The AI learning gpus (A100,
| H100 etc.) are not the best TFlop performers, but they have
| most vram. It seems that researchers found a sweet spot for
| neural network architectures that perform good on similar
| configurations, i.e. near real time (reading speed in LLMs).
| Once you bring those models to cpu, they might get
| performance bound again. Llama.cpp somehow illustrates that a
| bit, for bigger models you tend to wait a lot for the answer.
| I suspect the story would be similar with igpus
| hawski wrote:
| So possibly some basic iGPU (maybe even Intel) with lots of
| VRAM assigned could be enough?
| mrtksn wrote:
| It appears to generate 30 tokens/s on iPhone 14 pro:
| https://i.imgur.com/AWTXtGA.png
|
| But for some reason it dramatically slows down after a few
| messages
|
| Edit:
|
| Oh no, this one also gives lectures instead of answering
| questions.
|
| https://i.imgur.com/eiuGzK4.jpg
|
| I'm afraid, in near future the only organic content on the
| internet would be only the type of content that LLMs refuse to
| generate.
| ericlewis wrote:
| Tokenization is a probable issue here, the longer the context
| the longer the initial processing with llama I think. Possible
| tokenizer is not optimized.
| 29athrowaway wrote:
| Everything except OpenCL?
| thepra wrote:
| ...how to remove that "As an AI language model, I do not..."
| limit or self-censorship?
| vGPU wrote:
| Likely due to the model used. It looks like they're presenting
| this as a framework so it should be possible to substitute a
| different model in.
| amelius wrote:
| What surprises me is that the approaches to making cross-platform
| GPU computing work are so much focused on one specific use-case,
| ML.
|
| It's like someone builds a CPU with a floating point unit
| specifically aimed at CAD software. Then someone else comes and
| builds a floating point unit for physics simulation. Then someone
| else ...
|
| Can't we just get a generic compute model, and make that work
| everywhere? And don't we already have that, e.g. CUDA?
| UncleEntity wrote:
| > It's like someone builds a CPU with a floating point unit
| specifically aimed at CAD software. Then someone else comes and
| builds a floating point unit for physics simulation. Then
| someone else ...
|
| The history of GPGPU in a nutshell...
|
| There are a few "generic compute models" but no incentive for
| the GPU manufacturers to support them over their proprietary
| model. Everyone could natively support Cuda and Vulcan and
| Metal and OpenCL and SPIR-V and...think I'm forgetting one but
| you get the point.
| MereInterest wrote:
| While the use cases are tailed for machine learning, that isn't
| as much of a limitation as it sounds. The computationally-heavy
| portions of a machine-learning model are usually matrix
| multiplication and/or convolutions. The low-level operations
| could be combined into the training/evaluation of a machine-
| learning model, or could be combined into a physics simulation.
| That they are marketed as ML co-processors doesn't restrict
| their usage, just as a "graphics processing unit" isn't
| restricted to use for graphics.
| amelius wrote:
| Yeah, but that kind of marketing sucks to some extent because
| if they say they support A then as a consumer you don't know
| if they will support B now and in the future.
| dontreact wrote:
| What useful things are people able to do with the smaller more
| inaccurate models? I have a hard time understanding why I would
| build on top of this, rather than just the openaI API, since the
| performance is so much better.
| killthebuddha wrote:
| This is not a direct answer to your question, but performance
| is better in terms of the _quality_ of completions but not in
| terms of price, latency, or uptime.
| spudlyo wrote:
| I'm running this on my iPhone 13 Pro Max as part of the Test
| Flight beta, and it's interesting. I don't believe I've ever run
| anything that's ever pushed my phone this hard, and you can feel
| the heat. The text output performance is pretty inconsistent, it
| was very fast at first but slowed down considerably after a few
| answers. In terms of quality, it's prone to hallucinations, which
| is not unexpected based on the size and highly compressed nature
| of the model.
|
| "The marvel is not that the bear dances well, but that the bear
| dances at all."
| andrewaylett wrote:
| I'd suggest that _everything_ output by an LLM is a
| hallucination, but it often enough hallucinates something that
| resembles reality.
| junrushao1994 wrote:
| Thanks for sharing! It's definitely a bit painstaking to get a
| real-world LLM running at all on an iPhone due to memory
| constraint. It's also quite compute-intense as it has 7B
| parameters, but we are glad that it's generating texts at
| reasonable speed!
|
| The model we are using is a quantized Vicuna-7b, which I
| believe is one of the best open-sourced models. Hallucination
| is a problem to all LLMs, but I believe research on model side
| would gradually alleviate this problem :-)
| Fr0styMatt88 wrote:
| WizardLM-7b would be a fantastic model to try out as well.
| Though that might be out of date tomorrow (or already!) given
| how many models are being released at the moment :)
| kiratp wrote:
| You can do Stable Diffusion on your phone too, fully local.
|
| https://apps.apple.com/app/id6444050820
| david-gpu wrote:
| _> I don 't believe I've ever run anything that's ever pushed
| my phone this hard, and you can feel the heat. The text output
| performance is pretty inconsistent, it was very fast at first
| but slowed down considerably after a few answers._
|
| Those two events are causally related. The OS has to throttle
| down the CPU or else it will overheat and malfunction.
|
| It is one of the reasons why heavy number crunching is often
| performed on the cloud instead.
| trifurcate wrote:
| The slowdown here is going to be more about the increased
| context length than throttling.
| david-gpu wrote:
| Good point! Didn't think of that.
| capableweb wrote:
| In my experience, heavy number crunching is more suitable to
| run on dedicated machines, rather than virtualized "cloud"
| cores. More consistent performance, no noisy neighbors and
| cheaper in the long term.
| david-gpu wrote:
| "The cloud" simply means remote servers. Those servers may
| or may not contain application-specific hardware
| acceleration.
|
| Nowadays many cell phone application processors also have
| dedicated hardware to accelerate neural nets, but they will
| always be limited by thermal constraints.
| smoldesu wrote:
| If you can saturate it, sure. The average person is not
| going to be running 24/7 AI compute tasks though, so paying
| for a month of A100 dedi when you only need 10-20 minutes
| of compute/week is a complete waste.
| halflings wrote:
| > cheaper in the long term
|
| Citation needed :)) There are economies of scale and
| various optimizations that are just not possible with
| dedicated machines.
|
| Anecdotal evidence: a company I worked in had a dedicated
| DC with hundreds/thousands of machines that mostly ran SQL
| queries on petabytes of data (any query would take ~5-30
| minutes). Eye-watering budget and whole teams to maintain
| the cluster... They switched to GCP/BigQuery, got queries
| that ran in seconds at a fraction of the budget.
| Negitivefrags wrote:
| This anecdote is the opposite of the ones I usually hear.
|
| Economies of scale go in the other direction usually.
| slimsag wrote:
| The inverse is also true; I don't have to pay Google or
| AWS to deploy my software to dedicated consumer devices.
| The user already paid for it.
| [deleted]
| kiratp wrote:
| Why not let this be installed on Mac devices via test flight?
| crowwork wrote:
| There is a conda app that can be installed on macos
___________________________________________________________________
(page generated 2023-04-30 23:01 UTC)