[HN Gopher] Yarn-Mistral-7B-128k
       ___________________________________________________________________
        
       Yarn-Mistral-7B-128k
        
       Author : tosh
       Score  : 38 points
       Date   : 2023-11-11 19:46 UTC (3 hours ago)
        
 (HTM) web link (huggingface.co)
 (TXT) w3m dump (huggingface.co)
        
       | brucethemoose2 wrote:
       | This is super cool. Mistral is great, the full 128K context will
       | fit on cheap (24GB, maybe less) GPUs thanks to exLlamav2, and
       | they just pulled support into transformers instead of dropping
       | some weird proprietary code.
       | 
       | One critique of the graph though... I measured perplexity of
       | MistralLite myself, and its good out to 32K, but in the graph it
       | drops off where the base model drops:
       | https://huggingface.co/SciPhi/SciPhi-Mistral-7B-32k/discussi...
        
       | siquick wrote:
       | Is there any rough rules on what model sizes will work on various
       | RAM sizes on consumer machines?
        
         | washadjeffmad wrote:
         | For "full" context, typically:                   7B: 8GB (4GB
         | min for model only)         13B: 12GB (8GB min)         34B:
         | 28GB (18GB min)         70B: 60GB (38GB min)
         | 
         | Lower bpw quants and 8-bit cache size can reduce memory
         | requirements / expand context, but if you're sticking to
         | mainstream L1/L2 models, just go by that.
        
           | coder543 wrote:
           | For 128k context (even with a 7B model), I don't think 8GB is
           | nearly enough. I've heard there might be tricks to get it
           | under 24GB... but I haven't personally seen it under about
           | 28GB, IIRC. Definitely not heard of it being possible with
           | 8GB.
        
         | kiviuq wrote:
         | there's a calculator / estimator somewhere on Hugging Face that
         | can give you some approximate numbers based on the model you
         | select.
        
       | seydor wrote:
       | why does this require trust_remote_code
        
         | minimaxir wrote:
         | Because it requires shenanigans to get to the 128k context
         | window.
         | 
         | It's not malicious, you can view the code yourself in the .py
         | files: https://huggingface.co/NousResearch/Yarn-
         | Mistral-7b-128k/tre...
        
         | luke-stanley wrote:
         | You could also ask, why Pickle and not Safer Tensors or
         | Msgpack?
        
         | sp332 wrote:
         | I think any model type that is not built in to the huggingface
         | library needs that.
        
       | pram wrote:
       | Looks like you can already use it with ollama, nice
       | 
       | https://ollama.ai/library/yarn-mistral
        
         | wenc wrote:
         | Nice! I can finally run a model locally on my 16GB M2 Pro Mac
         | without a lot of futzing around with CUDA.
         | 
         | Ollama is very nice.
        
       ___________________________________________________________________
       (page generated 2023-11-11 23:01 UTC)