[HN Gopher] Show HN: Zipslicer, a library for loading LLM checkp...
___________________________________________________________________
Show HN: Zipslicer, a library for loading LLM checkpoints on
consumer hardware
This is a low-level opensource library I developed for my own use
and decided to share, as it makes it possible to process large
checkpoints of neural networks without renting high-RAM instances,
on a regular PC. It replaces torch.load() with a custom function
that produces a dictionary that materializes tensors on the fly.
Compared to other solutions it doesn't require sharding or re-
encoding checkpoints and uses them completely as-is. It is a
foundation to make it possible to run inference and compress
language models and other large models one layer at a time - in
principle, even one tensor at a time. I describe the rationale and
technical details of the library's design in the blogpost:
https://kir-gadjello.github.io/posts/zipslicer/
Author : kir-gadjello
Score : 23 points
Date : 2023-03-03 19:29 UTC (3 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| braindead_in wrote:
| Can it be used for Whisper?
| bootsmann wrote:
| This is a brilliant idea, I'll take this for a spin over the
| weekend. Thank you.
| asah wrote:
| how's performance? that's the usual issue with cutting up large
| models...
| Bjartr wrote:
| Has anyone tried this on the recently leaked LLAMA model?
| kir-gadjello wrote:
| Technically, this library is for loading a checkpoint one layer
| at a time and would work for any model you can load from a
| state_dict.
|
| To support the inference for the model of your choice, someone
| should write code that uses zipslicer to instantiate layers of
| the model while threading activations through active layers -
| and it will make economic sense if you use this code with large
| batch size in long-term computation. There is no magic trick
| yet to compute interactive sampling (think ChatGPT) without
| fitting the whole model in, at least, RAM+VRAM, better in VRAM.
|
| You could go slightly different route and use this non-
| interactive mode just one time to compress the weights down to
| int8, and this can, in turn, make the model runnable on your
| hardware in interactive mode without layer streaming from disk.
| Say, your model is 12B params and you have 16G RAM - it can
| work perfectly well if each weight takes 1 byte, but the
| weights are distributed in bf16/fp16 and your PC doesn't even
| support these so it instantiates the weights in fp32 - and you
| are out of RAM. Zipslicer can help here by allowing you to
| execute weight compression.
|
| While this "offline execution engine" or "parameter compression
| engine" is slightly beyond the scope of this library, I can
| write it as an example or as a separate library, and describe
| this in a blogpost. For the purpose of making an example I
| would prefer to work with permissively licensed models such as
| this one https://huggingface.co/Salesforce/codegen-6B-nl - but
| someone could easily fork the repo and adapt the code later to
| whatever model they need.
|
| It would be nice if people proposed desirable features in
| project's Github issues and I could see which features get
| votes.
| chrisMyzel wrote:
| How is the actual VRAM requirement calculated (says 175B on mid-
| range GPU). Do I need a max of one layer in VRAM and any more
| will just speed up inference?
___________________________________________________________________
(page generated 2023-03-03 23:01 UTC)