[HN Gopher] Open-Llama: Complete training pipeline for building ...
       ___________________________________________________________________
        
       Open-Llama: Complete training pipeline for building large language
       models
        
       Author : bayes-song
       Score  : 122 points
       Date   : 2023-05-14 01:21 UTC (21 hours ago)
        
 (HTM) web link (github.com)
 (TXT) w3m dump (github.com)
        
       | bayes-song wrote:
       | Check out this model trained using the Open-Llama project at
       | http://home.ustc.edu.cn/~sl9292 . This model is trained primarily
       | on English and Chinese, but also has capabilities in other
       | languages like Japanese and Korean. Now, let's dive into Open-
       | Llama. It's a truly open-source project for pre-training and
       | instruct-tuning AI models. One of the key features of this
       | project is its support for a wide range of model sizes, from 7B
       | to 65B parameters. What sets Open-Llama apart is the
       | incorporation of performance acceleration via xformers from
       | Llama, enabling 95% of the original Llama speed on the 65B
       | models. In fact, for the 7B models, Open-Llama's performance
       | surpasses the original Llama. By providing full access to the
       | codebase, we believe that Open-Llama will contribute greatly to
       | the advancement of open-source AI technologies. We invite
       | developers and researchers to join us on this exciting journey!
        
       | lhl wrote:
       | Namespace collisions are inevitable, especially w/ how fast-
       | moving the LLM space is right now, just wanted to point out that
       | besides this "Open-Llama" project (which looks really
       | interesting, and well documented in the Github repo), there is
       | also another group training "OpenLLaMA"
       | https://github.com/openlm-research/open_llama (which looks like
       | an effort by two Berkeley PhD students, https://www.haoliu.site/
       | and http://young-geng.xyz/ to reproduce LLaMA using the 1.2T
       | token Together RedPajama dataset. They've released up to a 300B
       | checkpoint so far.)
       | 
       | Feedback for /u/bayes-song - it'd be great to have a more info on
       | the model card on HF - right now it's unclear the parameter
       | count, # of total tokens you're planning on training on/how many
       | you've trained on so far. An Evaluation section (maybe using lm-
       | evaluation-harness) might be good as well?
        
         | pythux wrote:
         | To add to that, I believe the title of this submission is a
         | reference ("Open-Lamam: A "real" open-source project to train
         | LLM not just checkpoints") to this project you link, since they
         | did not (to my knowledge), release the code for the training or
         | detailed instructions to reproduce their experiment precisely,
         | only checkpoints.
        
       | marcopicentini wrote:
       | Sorry for newbie question. What's the advantage of retraining the
       | model versus using an already trained model through API (OPEN
       | AI). I understand the economic principle of "make or buy", but is
       | there else?
        
       | christkv wrote:
       | Feels like its still the area of wait and see as the space shakes
       | out. It would be great to be able to run our own models in some
       | near future for applications but the amount of hardware needed to
       | delivery service to a significant audience is pretty crazy. Right
       | now I don't see any way but to re-bill the cost with a markup to
       | end customers unless you have a giant pile of VC money that you
       | can light on fire.
        
         | quickthrower2 wrote:
         | I found running the model on rented hardware much more
         | expensive than ChatGPT. Might work ok for local sunk cost
         | hardware for those who game and don't crypto mine.
        
         | ingenieroariel wrote:
         | WebGPU would be a way to shift that cost back to each device.
         | RedPajama 3B could become useful for some tasks and run quite
         | fast on most hardware available. Then as users have better
         | computers, they can get access to better models?
        
           | snovv_crash wrote:
           | You'd have to have each user download a 3GB payload first.
           | For comparison, that's a good few hours of netflix at 1080p.
        
             | sroussey wrote:
             | Browsers and operating systems will eventually include
             | them.
             | 
             | But for now, you would need a good privacy reason to go
             | this route.
        
       | [deleted]
        
       ___________________________________________________________________
       (page generated 2023-05-14 23:02 UTC)