[HN Gopher] BLOOM: The largest open multilingual language model
       ___________________________________________________________________
        
       BLOOM: The largest open multilingual language model
        
       Author : georgehill
       Score  : 53 points
       Date   : 2022-07-12 09:58 UTC (13 hours ago)
        
 (HTM) web link (bigscience.huggingface.co)
 (TXT) w3m dump (bigscience.huggingface.co)
        
       | PoignardAzur wrote:
       | You can try the model on HuggingFace:
       | https://huggingface.co/bigscience/bloom
       | 
       | Though it requires an account, unlike other models on the hub (I
       | guess they're hoping to get new users from the hype of this
       | model).
       | 
       | Bloom looks pretty exciting. It's reportedly as performant as
       | GPT-3 (I haven't tested it enough to confirm, but what little
       | testing I did gave okay results), but the model is completely
       | open-source; if you can afford some cloud compute, you can just
       | upload the model on your preferred cloud provider and just
       | generate whatever you want from it, good or evil.
       | 
       | I'm expecting the coming 12 months to be pretty interesting for
       | text generation.
        
         | dekhn wrote:
         | If you create a hugginface account, can you download the model
         | and weights and run inference locally (I have a nvidia 3080 Ti
         | on a 32-core machine with 64GB of RAM)? Or do they just host
         | the inference server for you? IIUC the model design is open so
         | I assume the weights are too.
        
           | moyix wrote:
           | You can actually download the weights without an account:
           | git clone https://huggingface.co/bigscience/bloom
           | 
           | You will need Git LFS and ~330GB of free space though.
           | 
           | It is possible to run inference locally, even without very
           | much RAM/VRAM (though it will be _very_ slow). Get the most
           | recent versions of  "transformers" and "accelerate" from git,
           | and then:                   import torch         from
           | transformers import AutoTokenizer, AutoModelForCausalLM
           | tokenizer = AutoTokenizer.from_pretrained('./bloom')
           | inputs = tokenizer("Hacker News is", return_tensors="pt")
           | model = AutoModelForCausalLM.from_pretrained('./bloom',
           | device_map="auto", offload_folder='offload',
           | torch_dtype=torch.bfloat16)                  output =
           | model.generate(inputs["input_ids"].to(0), min_length=30,
           | max_length=30, do_sample=True)
           | print(tokenizer.decode(output[0].tolist()))
           | 
           | After the model is loaded you can also look at where it put
           | each part of the model (GPU, CPU RAM, or on disk):
           | print(model.hf_device_map)
           | 
           | Here's a screenshot of it in action on my machine, which has
           | 256GB RAM and 2xA6000 GPUs:
           | 
           | https://twitter.com/moyix/status/1546961979566325760
        
             | dekhn wrote:
             | Thanks, that's what I was looking for (double thanks for
             | the example inference code). I'll queue up the download and
             | try to get the dependencies working.
             | 
             | Comment 1: git-lfs is now close to saturating my gigabit
             | ethernet connection, around 990Mb/sec
        
       ___________________________________________________________________
       (page generated 2022-07-12 23:01 UTC)