[HN Gopher] AirLLM enables 8GB MacBook run 70B LLM
       ___________________________________________________________________
        
       AirLLM enables 8GB MacBook run 70B LLM
        
       Author : SlavikCA
       Score  : 51 points
       Date   : 2023-12-28 05:34 UTC (17 hours ago)
        
 (HTM) web link (github.com)
 (TXT) w3m dump (github.com)
        
       | mdrzn wrote:
       | Whoa, I don't understand enough to figure out if this is real and
       | scalable or not, but if this is true it's a HUGE step forward.
       | Can't wait to try and run a 70b LLM on my 32GB RAM desktop w/
       | Windows.
        
       | rahimnathwani wrote:
       | The acknowledgements section in the README links to notebook,
       | which is where OP sourced the techniques:
       | 
       | https://www.kaggle.com/code/simjeg/platypus2-70b-with-wikipe...
       | 
       | The notebook might be an easier read than the repo, but I haven't
       | read either yet.
       | 
       | EDIT: It's very slow, according to the comments in this thread by
       | people who tried it: https://github.com/oobabooga/text-
       | generation-webui/issues/47...
        
       | erikaww wrote:
       | This has mixtral support! Can't wait to see the next wave of
       | local MOE models. Perhaps cheap fast and local GPT-4 performance
       | is not too far off.
        
       | great_psy wrote:
       | I did not dig too deep in the technicalities of it, but is there
       | anything that would stop openAI from also implementing something
       | like this ?
       | 
       | Presumably any advances open source community makes towards
       | running on cheap hardware, will also massively benefit the big
       | guys.
        
       | jug wrote:
       | Could 2024 become a crisis for commercial AI?
       | 
       | 1. We're only barely getting started with free MOE models and
       | Mistral has already impressed.
       | 
       | 2. Cloud AI is a poor fit for corporate use at least in EU due to
       | GDPR, the NIS directive and more. You really dont want to exit
       | the EU in where the data processing takes place.
       | 
       | 3. There are indications of diminishing returns in LLM
       | performance, where a one year later shot at it from Google,
       | despite massive resources in terms of both experts and data set,
       | still doesn't have Gemini Pro clearly surpass GPT 3.5 and Ultra
       | probably not GPT 4. Meanwhile, competition like Mistral is
       | closing in.
       | 
       | 4. The NY Times lawsuit seems like a harbinger for what is to
       | become a theme for AI companies in 2024. Open collaborations are
       | harder to target as legal entities and there is not nearly as
       | much money to gain if you win.
       | 
       | All this points toward a a) convergence of performance that will
       | b) be in the benefit of open models to me.
       | 
       | Interesting times anyway especially as we are STILL only getting
       | started.
        
         | aaomidi wrote:
         | I feel like AI was a crisis for commercial AI.
        
       | ceeam wrote:
       | At what SSD wear rate?
        
       ___________________________________________________________________
       (page generated 2023-12-28 23:02 UTC)