[HN Gopher] State-of-the-Art Chatbot, Vicuna-7B, now runs on Mac...
___________________________________________________________________
State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU
acceleration
Author : weichiang
Score : 61 points
Date : 2023-04-06 20:45 UTC (2 hours ago)
(HTM) web link (twitter.com)
(TXT) w3m dump (twitter.com)
| bawana wrote:
| MacBook with M1 chip here.python installed with homebrew tried to
| install with: pip install fschat
|
| then tried to run it with: python3 -m fastchat.serve.cli --model
| -name vicuna-7b --device mps --load-8bit
|
| got this:
|
| traceback (most recent call last): File "<frozen runpy>", line
| 198, in _run_module_as_main File "<frozen runpy>", line 88, in
| _run_code File "/opt/homebrew/lib/python3.11/site-
| packages/fastchat/serve/cli.py", line 9, in <module> from
| transformers import AutoTokenizer, AutoModelForCausalLM,
| LlamaTokenizer ModuleNotFoundError: No module named
| 'transformers'
|
| so I did this: pip install transformers command
|
| tried again:
|
| python3 -m fastchat.serve.cli --model -name vicuna-7b --device
| mps --load-8bit
|
| got:
|
| Traceback (most recent call last): File
| "/opt/homebrew/lib/python3.11/site-
| packages/transformers/utils/import_utils.py", line 1126, in
| _get_module return importlib.import_module("." + module_name,
| self.__name__)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/
| opt/homebrew/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framew
| ork/Versions/3.11/lib/python3.11/importlib/__init__.py", line
| 126, in import_module return _bootstrap._gcd_import(name[level:],
| package, level)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File
| "<frozen importlib._bootstrap>", line 1206, in _gcd_import File
| "<frozen importlib._bootstrap>", line 1178, in _find_and_load
| File "<frozen importlib._bootstrap>", line 1128, in
| _find_and_load_unlocked File "<frozen importlib._bootstrap>",
| line 241, in _call_with_frames_removed File "<frozen
| importlib._bootstrap>", line 1206, in _gcd_import File "<frozen
| importlib._bootstrap>", line 1178, in _find_and_load File
| "<frozen importlib._bootstrap>", line 1149, in
| _find_and_load_unlocked File "<frozen importlib._bootstrap>",
| line 690, in _load_unlocked File "<frozen
| importlib._bootstrap_external>", line 940, in exec_module File
| "<frozen importlib._bootstrap>", line 241, in
| _call_with_frames_removed File
| "/opt/homebrew/lib/python3.11/site-
| packages/transformers/models/__init__.py", line 15, in <module>
| from . import ( File "/opt/homebrew/lib/python3.11/site-
| packages/transformers/models/mt5/__init__.py", line 29, in
| <module> from ..t5.tokenization_t5 import T5Tokenizer File
| "/opt/homebrew/lib/python3.11/site-
| packages/transformers/models/t5/tokenization_t5.py", line 26, in
| <module> from ...tokenization_utils import PreTrainedTokenizer
| File "/opt/homebrew/lib/python3.11/site-
| packages/transformers/tokenization_utils.py", line 26, in
| <module> from .tokenization_utils_base import ( File
| "/opt/homebrew/lib/python3.11/site-
| packages/transformers/tokenization_utils_base.py", line 74, in
| <module> from tokenizers import AddedToken File
| "/opt/homebrew/lib/python3.11/site-
| packages/tokenizers/__init__.py", line 80, in <module> from
| .tokenizers import ( ImportError:
| dlopen(/opt/homebrew/lib/python3.11/site-
| packages/tokenizers/tokenizers.cpython-311-darwin.so, 2): no
| suitable image found. Did find:
| /opt/homebrew/lib/python3.11/site-
| packages/tokenizers/tokenizers.cpython-311-darwin.so: mach-o, but
| wrong architecture /opt/homebrew/lib/python3.11/site-
| packages/tokenizers/tokenizers.cpython-311-darwin.so: mach-o, but
| wrong architecture
|
| The above exception was the direct cause of the following
| exception:
|
| Traceback (most recent call last): File "<frozen runpy>", line
| 198, in _run_module_as_main File "<frozen runpy>", line 88, in
| _run_code File "/opt/homebrew/lib/python3.11/site-
| packages/fastchat/serve/cli.py", line 9, in <module> from
| transformers import AutoTokenizer, AutoModelForCausalLM,
| LlamaTokenizer File "<frozen importlib._bootstrap>", line 1231,
| in _handle_fromlist File "/opt/homebrew/lib/python3.11/site-
| packages/transformers/utils/import_utils.py", line 1116, in
| __getattr__ module =
| self._get_module(self._class_to_module[name])
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File
| "/opt/homebrew/lib/python3.11/site-
| packages/transformers/utils/import_utils.py", line 1128, in
| _get_module raise RuntimeError( RuntimeError: Failed to import
| transformers.models.auto because of the following error (look up
| to see its traceback): dlopen(/opt/homebrew/lib/python3.11/site-
| packages/tokenizers/tokenizers.cpython-311-darwin.so, 2): no
| suitable image found. Did find:
| /opt/homebrew/lib/python3.11/site-
| packages/tokenizers/tokenizers.cpython-311-darwin.so: mach-o, but
| wrong architecture /opt/homebrew/lib/python3.11/site-
| packages/tokenizers/tokenizers.cpython-311-darwin.so: mach-o, but
| wrong architecture
| zhisbug wrote:
| No llama.cpp nor any compilation complexity. Run with two Python
| commands!
| sottol wrote:
| Did they release the merged weights, yet? I'd love to try this
| model.
|
| Afaict from the docs, you still need to request the original
| Llama weights from Meta (or get ahold of them another way),
| then apply the diff-weights requiring 60GB RAM?
| superkuh wrote:
| I think you have it backwards. The python (ie, huggingface,
| etc) implementations of transformers are the complex ones with
| dependency hell so bad even there's even a layer of package
| manager / env hell. This version of fastchat (there's 2)
| required a particular commit of huggingface libs for quite a
| while. Something that only changed recently. And it'll happen
| again in the future. Python just hides this complexity... until
| it doesn't. Like beautiful but rapidly rotting fruit.
|
| llama.cpp will remain a single two line project (git clone
| https://github.com/ggerganov/llama.cpp, make -j) that will
| compile easily and run on anything. No external deps to pin to
| a particular commit (that will only have a lifetime of some
| months) as things change rapidly.
|
| That said, the changes in the ggml weights format the last 2
| weeks were annoying, but now that the mmap-style weights are
| settled on it should be less converting. In that sense
| huggingface wins, it only has two incompatible weights formats.
| llama.cpp's ggml has had 3.
| sterlind wrote:
| I've spent the past couple days packaging an LLM playground
| environment as a Nix expression. it's been pure hell.
|
| also nice to see you again, superkuh. I frequented your IRC
| channel about a decade ago.
| siraben wrote:
| Have you been successful in getting the LLM playground up
| with Nix?
| sterlind wrote:
| yes, almost! I used poetry2nix and grafted a bunch of
| overrides to fix the torch-2.0 build, and I just got cuda
| working with it. I'm testing triton now. I'll submit my
| PR to poetry2nix so watch that space if you want it.
| superkuh wrote:
| Using nix and then complaining about having to set up your
| compilation environment libs/etc is kind of like sticking a
| rod in your bike's wheel spokes and complaining about
| crashing. Don't give up on the idea of system libraries
| (ie, use nix) and this doesn't happen.
|
| Also, hi? I don't recall you by that nick but the internet
| is a small place sometimes.
| sterlind wrote:
| oh, I'm very aware that I've brought this upon myself,
| but I'm sticking out for the greater good (and
| stubbornness.)
|
| specifically, I'm trying to benchmark a bunch of
| different GPU configurations on different workloads on
| vast.ai, which uses Docker containers. I abhor
| Dockerfiles and my experience building containers with
| nix has been pleasant, so that's what I'm doing and why.
| fortunately I think I'm getting past the learning curve.
|
| did our channel survive the demise of freenode? I was
| andares, I think I used to be annoying but I've gotten
| better.
| superkuh wrote:
| Ah. Hi! Yes. We still exist in the same place but on
| libera now.
| zhisbug wrote:
| no, the requirement on a particular HF commit has been fixed.
| It is no longer needed.
| superkuh wrote:
| Right. That particular problem has been fixed. But the fact
| that it was needed indicates it will happen again. It
| exposes the underlying complexity of the huggingface
| transformer stack. It's wonderful code, don't get me wrong.
| It's just the furthest thing possible from the least
| complex.
| zhisbug wrote:
| it is really a matter of having faith on pytorch (or JAX)
| or on third-party cross-platform supports like llama-cpp.
| Apparently pytorch reduces a lot of complexity and grows
| extremely faster on cross-platform supports.
|
| And, PyTorch does so well on GPUs!
| Casteil wrote:
| This has been my experience so far as well. GPT4All feels
| pretty fragile with all its dependencies.
| youssefabdelm wrote:
| So far I think what these models lack is memory of people and
| other things. Especially if not as popular. And probably a ton
| more.
|
| E.g. try asking it "Who is Tyler Volk?"
|
| Then try asking GPT-4 "Who is Tyler Volk?"
|
| Then check who he is online.
| circuit10 wrote:
| Probably because the parameter count is way lower so it's less
| able to memorize things
| psychphysic wrote:
| The language is also quite unnatural feeling.
|
| Neat none the less but hardly a standout in my opinion.
|
| Everything is state of the art at the moment I guess so can't
| criticise that too much.
| Casteil wrote:
| Anyone here who's used both this and GPT4All? Any thoughts/input
| on how they compare?
| weichiang wrote:
| I asked GPT4All one of Vicuna's benchmark questions:
|
| "What if the Internet had been invented during the Renaissance
| period?"
|
| Check out their responses: https://imgur.com/a/mPrdZ1W More
| questions here: https://vicuna.lmsys.org/eval/
|
| Note: not an apple-to-apple comparison but that's the model
| checkpoint I found on their git repo.
| superkuh wrote:
| My one take away after playing with both chat mode and text
| completion modes is that gpt4all 7B 4bit stays on the chat
| rails (doesn't start taking the role of the user, or spewing
| fine tuning boilerplate) much better than vicuna 7B 4bit. In
| text completion they're about the same but I'd still prefer the
| vanilla llama 7B in that case.
|
| There are a couple versions of gpt4all fine-tuned llama 7B and
| my favorite is the unfiltered one (gpt4all-lora-unfiltered-
| quantized.bin). https://github.com/nomic-ai/gpt4all#try-it-
| yourself
| zhisbug wrote:
| Lmsys hasn't released any official 4-bit version. It might be
| a better idea to wait for the official 4-bit version. But it
| is interesting to learn that the third-party 4bit version has
| performance degeneration.
| superkuh wrote:
| Lmsys hasn't released _any_ official weights for anything.
| They 've released "deltas" and other people have applied
| those deltas to the appropriate llama weights and done the
| quantization.
|
| I reject your premise that the 8 to 4 bit quantization is
| the cause of the vicuna fine-tuned llamas very average
| performance though. This hasn't been the case for any of
| the other 8 to 4 bit quantizations. It would be a unique
| outlier. And so I don't think this is the "cause" here.
| zhisbug wrote:
| My point is that I am not aware of any official 4-bit
| quantization version (delta or weights) by lmsys so it
| might too early to draw your conclusion that vicuna
| finetuned llamas degenerates a lot of performance at 4
| bit but others are fine.
| zhisbug wrote:
| And I think the problem of taking the roles of users in
| vicuna is caused by this bug: https://github.com/lm-
| sys/FastChat/commit/1bb234265d16bdfd50...
|
| which has been fixed recently.
|
| Lmsys are launching new training jobs after this patch,
| please stay tuned.
| superkuh wrote:
| Nah, I don't use huggingface transformers to run
| inference with the vicuna model. I use llama.cpp. But I
| do appreciate the tip.
| wejick wrote:
| Seems like llama derived model are flourishing. However with
| llama is licensed as academic only and noncommercial model, what
| is the path for bringing this to production of for profit
| purpose?
|
| I certainly interested doing so.
| alwayslikethis wrote:
| Copilot style. Train a distilled model based on it, and now
| it's a new model unencumbered by copyright.
| ReptileMan wrote:
| The Silicon Valley ethos has always been - do it first worry
| about legality later. If you go bust - nobody will care. If you
| become small - you will be ignored. If you go big - lawyers
| will figure something out to cut a deal.
| JohnFen wrote:
| That is a thoroughly bankrupt ethos that should be denounced
| every time it pops up. It is literally condoning criminality.
| avereveard wrote:
| The methodology for alpaca has proven powerful and it's being
| applied to model with better licensing. It's hard to track
| lineage, but I think openassistant models are the most
| permissive at the moment, they use a openly sourced set of data
| to build an instruct model on top of phiia, which itself is a
| gptneox trained on a duplicated version of the famous the pile
| dataset.
|
| The problem is verifying the licensing claims for these
| composed solutions is becoming exceedingly hard.
| tric wrote:
| Why is there so much focus on running GPT models on Mac OS? Is
| there something special about Apple's new chip, or Mac OS?
| matwood wrote:
| The shared ram and neural engine make for an
| interesting/powerful platform if people are willing to port to
| it.
| wmf wrote:
| Apple's unified memory should allow running large models like
| 65B that will not fit on a consumer GPU, but mostly I see
| people talking about the smaller 7B sizes that can run
| anywhere.
| 19h wrote:
| Unified memory allows both CPU and GPU to use the same memory,
| effectively giving a MacBook with 96GB of memory 96GB of VRAM
| (minus OS overhead obv).
| nickthegreek wrote:
| I can run the 30b 4bit model on my m2 air that has 24gb of ram.
___________________________________________________________________
(page generated 2023-04-06 23:01 UTC)