[HN Gopher] llama.cpp: Roadmap May 2023
       ___________________________________________________________________
        
       llama.cpp: Roadmap May 2023
        
       Author : tosh
       Score  : 77 points
       Date   : 2023-04-30 07:55 UTC (15 hours ago)
        
 (HTM) web link (github.com)
 (TXT) w3m dump (github.com)
        
       | submeta wrote:
       | Can I run this on a intel Mac? And can I have a non censored
       | version of ChatGPT with this?
        
         | orost wrote:
         | There is nothing out there that quite matches ChatGPT quality
         | but you can get a similar kind of experience by running an
         | instruction-tuned derivative of LLaMA with llama.cpp. Try
         | something like vicuna-13b-free or oasst-sft-6-llama-30b on for
         | size. The former is trained on ChatGPT output but with refusals
         | (the censorship) removed, which seems to have mostly worked.
         | The latter is safety-trained, but from what I've heard (I need
         | a hardware upgrade before I can run 30B models), much more
         | mildly than ChatGPT.
         | 
         | It should run on whatever as long as you have enough memory.
         | How much exactly depends on the quantization mode chosen (it's
         | a quality-memory-speed tradeoff), but you should expect to need
         | between 0.5 and 1GB of memory per 1B parameters in the model.
        
       | sourcecodeplz wrote:
       | I'm on Windows 10, 3.5GHz AMD CPU (dual core, iGPU), 8gb RAM and
       | I get about 3 tokens per second with the smallest model (ggml-
       | alpaca-7b-q4).
        
       | exogeny wrote:
       | This code really whips the llama's ass.
        
         | varelse wrote:
         | [dead]
        
         | orra wrote:
         | Underappreciated Winamp reference.
        
       ___________________________________________________________________
       (page generated 2023-04-30 23:01 UTC)