https://gist.github.com/rain-1/8cc12b4b334052a21af8029aa9c4fafc

Skip to content
 
[                    ]
All gists Back to GitHub Sign in Sign up
 
Sign in Sign up
{{ message }}

Instantly share code, notes, and snippets.

@rain-1

rain-1/llama-home.md

Last active May 14, 2023 22:42

  * Star 177
  * Fork 13

Star
Code Revisions 11 Stars 177 Forks 13
Embed
What would you like to do?
">Embed Embed this gist in your website.
Share Copy sharable link for this gist.
Clone via HTTPS Clone with Git or checkout with SVN using the
repository's web address.
 
Learn more about clone URLs
[<script src="https:/]
 
Download ZIP
How to run Llama 13B with a 6GB graphics card
Raw
llama-home.md

This worked on 14/May/23. The instructions will probably require
updating in the future.

    llama is a text prediction model similar to GPT-2, and the
    version of GPT-3 that has not been fine tuned yet. It is also
    possible to run fine tuned versions (like alpaca or vicuna with
    this. I think. Those versions are more focused on answering
    questions)

Note: I have been told that this does not support multiple GPUs. It
can only use a single GPU.

It is possible to run LLama 13B with a 6GB graphics card now! (e.g. a
RTX 2060). Thanks to the amazing work involved in llama.cpp. The
latest change is CUDA/cuBLAS which allows you pick an arbitrary
number of the transformer layers to be run on the GPU. This is
perfect for low VRAM.

  * Clone llama.cpp from git, I am on commit
    08737ef720f0510c7ec2aa84d7f70c691073c35d.
      + git clone https://github.com/ggerganov/llama.cpp.git
      + cd llama.cpp
      + pacman -S cuda make sure you have CUDA installed
      + make LLAMA_CUBLAS=1
  * Use the link at the bottom of the page to apply for research
    access to the llama model: https://ai.facebook.com/blog/
    large-language-model-llama-meta-ai/
  * Set up a micromamba environment to install cuda/python pytorch
    stuff in order to run the conversion scripts. Install some
    packages:
      + export MAMBA_ROOT_PREFIX=/path/to/where/you/want/mambastuff/
        stored
      + eval "$(micromamba shell hook --shell=bash)"
      + micromamba create -n mymamba
      + micromamba activate mymamba
      + micromamba install -c conda-forge -n mymamba pytorch
        transformers sentencepiece
  * Perform the conversion process: (This will produce a file called
    ggml-model-f16.bin)
      + python convert.py ~/ai/Safe-LLaMA-HF-v2\ \(4-04-23\)/
        llama-13b/
  * Then quantize that to a 4bit model:
      + ./quantize ~/ai/Safe-LLaMA-HF-v2\ \(4-04-23\)/llama-13b/
        ggml-model-f16.bin ~/ai/Safe-LLaMA-HF-v2\ \(4-04-23\)/
        llama-13b/ggml-model-13b-q4_0-2023_14_5.bin q4_0 8
  * Create a prompt file in:
      + prompt.txt
  * Run it:
      + ./main -ngl 18 -m ~/ai/Safe-LLaMA-HF-v2\ \(4-04-23\)/
        llama-13b/ggml-model-13b-q4_0-2023_14_5.bin -f prompt.txt -n
        2048

This uses about 5.5GB of VRAM on my 6GB card. If you have more VRAM,
you can increase the number -ngl 18 to -ngl 24 or so, up to all 40
layers in llama 13B. It will run faster if you put more layers into
the GPU. The 7B model works with 100% of the layers on the card.

Timings for the models:

13B:

llama_print_timings:        load time =  5690.77 ms
llama_print_timings:      sample time =  1023.87 ms /  2048 runs   (    0.50 ms per token)
llama_print_timings: prompt eval time = 36694.62 ms /  1956 tokens (   18.76 ms per token)
llama_print_timings:        eval time = 644282.27 ms /  2040 runs   (  315.82 ms per token)
llama_print_timings:       total time = 684789.56 ms

7B:

llama_print_timings:        load time = 41708.38 ms
llama_print_timings:      sample time =    88.51 ms /   128 runs   (    0.69 ms per token)
llama_print_timings: prompt eval time =  2971.75 ms /    14 tokens (  212.27 ms per token)
llama_print_timings:        eval time =  9097.33 ms /   127 runs   (   71.63 ms per token)
llama_print_timings:       total time = 50931.74 ms

Here is the text it generated from my prompt:

     Nietzsche's Noon

    In Friedrich Nietzsche's Thus Spoke Zarathustra (1885), this
    concept of noon is expanded upon as a whole:

    "Zarathustra saw that the light of the world was now becoming
    stronger. 'The sun is at its meridian,' he said, 'it has reached
    its noontide and will begin to decline.'"

    As time progresses in this noon, so does our ability to perceive
    and interact with this external world: as a result of the present
    state we are given by the organic forms of space and time, this
    can only lead us towards suffering. Nietzsche sees that when we
    are at our noontide, we must realize how the sun has reached its
    highest point before it begins to fall down from its position:
    and we must understand that as a whole, our bodies are determined
    by something outside of ourselves - and that this always leads to
    more suffering within.

     Nietzsche's Midday

    Nietzsche expands upon the concept of noon in his book The Gay
    Science (1882), where he says:

    "You want to learn how to read? Here is a short lesson for
    beginners. You must take hold of a word by its smooth or rough
    side; then, like the spider, you must spin out of it a web of
    definitions which will entrap every correct meaning that floats
    into view. Or again: you must take the word for a sleigh ride
    across country, over hedges and ditches, forests and glades, in
    short, you must drive the word home through all manner of
    weather."

    Here we see Nietzsche expand upon his concept of noon to include
    our ability to define what is right or wrong - it is only because
    we have this inherent sense that allows us to distinguish between
    two points.

     Nietzsche's Twilight

    Nietzsche expands upon the concept of twilight in The Gay Science
    (1882), where he says:

    "The man who is a 'philosopher' only by accident, but is, let us
    say, also a sculptor or painter - what does he then do? He does
    not make his thoughts subservient to the world; rather, he forces
    the world to serve as a pedestal and bearer for his thoughts."

    Here we see Nietzsche's concept of night begin to expand past an
    emotional state. We begin to see that night becomes more than
    just our inability to think clearly: it is now a worldview, one
    which he claims is best exemplified by the artist.

     Nietzsche's Midnight

    In Twilight of the Idols (1889), we find Nietzsche's
    conceptualization of night reaching its zenith:

    "Everything ordinary, everyday, common - in fact, everything that
    exists today has become dangerous; it is not innocent as was
    everything yesterday. For the most terrible thoughts have
    penetrated everywhere and even into the deepest sleep - thoughts
    which are awake, active, and powerful."

@StrumykTomira
Copy link

StrumykTomira commented May 14, 2023

Hi! It is possible to use it on more than one card in the same time?
Ie. i have 5 cards with 6GB of VRAM on each. I mean, it is possible
to merge the VRAM/layers of all of this cards?

Sorry, something went wrong.

@rain-1
Copy link
Author

rain-1 commented May 14, 2023

    Hi! It is possible to use it on more than one card in the same
    time? Ie. i have 5 cards with 6GB of VRAM on each. I mean, it is
    possible to merge the VRAM/layers of all of this cards?

To be honest, I have no idea. I have heard that dual GPU can work!
but I only have one GPU so that's all I can test with. If anybody
experiments with this I'd love to hear how it goes. I suppose you can
watch nvtop while it's running to see if it utilizes both cards.

Sorry, something went wrong.

@whjms
Copy link

whjms commented May 14, 2023

How does the performance compare to using llama.cpp on CPU only?

Sorry, something went wrong.

@rain-1
Copy link
Author

rain-1 commented May 14, 2023 *
edited

    How does the performance compare to using llama.cpp on CPU only?

There are a couple graphs here: https://www.reddit.com/gallery/
13h7cqe this is reporting 2x to 10x speedup for half on GPU. and 40x
or 7x for all of it on GPU.

This will be completely dependent on peoples setup, but for my CPU/
GPU combo and running 18 out of the 40 layers I got:

GPU:

llama_print_timings:        load time =  5799.77 ms
llama_print_timings:      sample time =   189.19 ms /   394 runs   (    0.48 ms per token)
llama_print_timings: prompt eval time =  8150.29 ms /   414 tokens (   19.69 ms per token)
llama_print_timings:        eval time = 120266.65 ms /   392 runs   (  306.80 ms per token)
llama_print_timings:       total time = 131062.07 ms


llama_print_timings:        load time =  6028.92 ms
llama_print_timings:      sample time =   982.32 ms /  2048 runs   (    0.48 ms per token)
llama_print_timings: prompt eval time = 36915.54 ms /  1956 tokens (   18.87 ms per token)
llama_print_timings:        eval time = 632517.50 ms /  2040 runs   (  310.06 ms per token)
llama_print_timings:       total time = 673192.05 ms



CPU:


llama_print_timings:        load time =  5689.34 ms
llama_print_timings:      sample time =   113.70 ms /   239 runs   (    0.48 ms per token)
llama_print_timings: prompt eval time =  3995.84 ms /   157 tokens (   25.45 ms per token)
llama_print_timings:        eval time = 108059.76 ms /   238 runs   (  454.03 ms per token)
llama_print_timings:       total time = 113920.11 ms



llama_print_timings:        load time =  5733.33 ms
llama_print_timings:      sample time =   915.47 ms /  1916 runs   (    0.48 ms per token)
llama_print_timings: prompt eval time = 41703.82 ms /  1956 tokens (   21.32 ms per token)
llama_print_timings:        eval time = 877239.06 ms /  1908 runs   (  459.77 ms per token)
llama_print_timings:       total time = 922038.11 ms

  * 2.1 tokens per second CPU (feels incredibly slow to watch)
  * 3.2 tokens per second GPU.

Sorry, something went wrong.

@nicewrld
Copy link

nicewrld commented May 14, 2023

For anyone running this in WSL that hasn't set up CUDA:

Get the installation instructions from https://developer.nvidia.com/
cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=
WSL-Ubuntu&target_version=2.0&target_type=deb_local
or, if you're daring, here are the ones I used:

wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin
sudo mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/12.1.1/local_installers/cuda-repo-wsl-ubuntu-12-1-local_12.1.1-1_amd64.deb
sudo dpkg -i cuda-repo-wsl-ubuntu-12-1-local_12.1.1-1_amd64.deb
sudo cp /var/cuda-repo-wsl-ubuntu-12-1-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update
sudo apt-get -y install cuda

I had an error running make, had to add cuda to path (this works bc /
usr/local/cuda is symlinked to whatever version you get)

export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}$
export LD_LIBRARY_PATH=/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

sources:

installing cuda toolkit
https://learn.microsoft.com/en-us/windows/ai/directml/gpu-cuda-in-wsl
https://docs.nvidia.com/cuda/wsl-user-guide/index.html#
getting-started-with-cuda-on-wsl

make issue
https://askubuntu.com/questions/885610/
nvcc-version-command-says-nvcc-is-not-installed

Sorry, something went wrong.

@nicewrld
Copy link

nicewrld commented May 14, 2023

Here are some timings from inside of WSL on a 3080 Ti + 5800X:

llama_print_timings:        load time =  4783.06 ms
llama_print_timings:      sample time =   990.63 ms /  2048 runs   (    0.48 ms per token)
llama_print_timings: prompt eval time = 15378.98 ms /  2391 tokens (    6.43 ms per token)
llama_print_timings:        eval time = 165769.76 ms /  2039 runs   (   81.30 ms per token)
llama_print_timings:       total time = 185387.49 ms

Sorry, something went wrong.

@rain-1
Copy link
Author

rain-1 commented May 14, 2023 *
edited

Thanks so much for sharing that information @nicewrld !

81.30 ms per token, nice!

Sorry, something went wrong.

@sashokbg
Copy link

sashokbg commented May 14, 2023

For some reason my GPU activity stays at 0 even though I set "-ngl
18" and I entered a long prompt (over 512). Do you have any ideas ?

Running on linux CPU x84_64

Sorry, something went wrong.

@rain-1
Copy link
Author

rain-1 commented May 14, 2023

That should be all the conditions fulfilled. I am not sure what else
could be stopping it from utilizing GPU.

Sorry, something went wrong.

Sign up for free to join this conversation on GitHub. Already have an
account? Sign in to comment

Footer

 (c) 2023 GitHub, Inc.

Footer navigation

  * Terms
  * Privacy
  * Security
  * Status
  * Docs
  * Contact GitHub
  * Pricing
  * API
  * Training
  * Blog
  * About

You can't perform that action at this time.
You signed in with another tab or window. Reload to refresh your
session. You signed out in another tab or window. Reload to refresh
your session.