[HN Gopher] Hyperparameter Optimization for LLMs via Scaling Laws
       ___________________________________________________________________
        
       Hyperparameter Optimization for LLMs via Scaling Laws
        
       Author : Lindizz
       Score  : 72 points
       Date   : 2023-06-06 10:43 UTC (12 hours ago)
        
 (HTM) web link (arxiv.org)
 (TXT) w3m dump (arxiv.org)
        
       | tartakovsky wrote:
       | What's better, train a model with 10X parameters once on some
       | default hyperparameter setting or to search for a good
       | hyperparameter configuration by training on X parameters 10
       | times? While I'm at it, how many LLMs of the size of GPT3 were
       | trained until they landed on the capability of GPT3? How much of
       | this is dependent on the data, or do good settings transcend the
       | type of text that a model is trying to train on?
        
         | samuellll wrote:
         | The questions you raise are very interesting. My question would
         | be, where does the default hyperparameter configuration come
         | from? Additionally, does there exist one hyperparameter
         | configuration that performs well on all tasks?
        
         | albert94 wrote:
         | The proposed idea here is different.
         | 
         | You can train several smaller models with different
         | hyperparameters with dynamic budgets, i.e. bad configurations
         | are trained for only few epochs, and good ones for more epochs.
         | Once you find a good hyperparameter configuration for the
         | small-scale model, then you train the large model with that
         | configuration.
         | 
         | What is being shown is that the overhead of doing
         | hyperparameter optimization at a small scale, is comparable to
         | a single optimization at the largest scale.
         | 
         | Overall, the idea looks very cool.
        
       | mlminer wrote:
       | Really nice, it is very useful now with so many new models and
       | datasets for NLP
        
       | jackylupino23 wrote:
       | Very interesting. Wondering what is the state of the art in
       | Hyperparameter Optimization at the moment. Does this method apply
       | to all Deep Learning systems?
        
         | rch wrote:
         | I just got back into hyperopt a couple weeks ago. It's easy
         | enough and worked for me, but I was thinking there had to be
         | some new things I'm not aware of.
        
         | blackbear_ wrote:
         | For a general overview, this could be a good starting point
         | [1]. As for deep learning, you may wanna start from here [2],
         | but I personally had good results with Hyperband [3] for DL.
         | 
         | [1]
         | https://wires.onlinelibrary.wiley.com/doi/full/10.1002/widm....
         | 
         | [2] https://github.com/google-research/tuning_playbook
         | 
         | [3] https://jmlr.csail.mit.edu/papers/v18/16-558.html
        
         | mlminer wrote:
         | I think so, they apply it to Computer Vision datasets as well
        
         | gillesjacobs wrote:
         | Hyperband [1] has been my go-to hyperparam optimization method
         | over the past few years. Handily beats Bayesian search wherever
         | I applied it, also implemented in most frameworks.
         | 
         | 1. https://arxiv.org/abs/1603.06560
        
           | Lindizz wrote:
           | The work compares against Hyperband and the new method is
           | significantly better (Figure 2, Hypothesis 2).
        
       ___________________________________________________________________
       (page generated 2023-06-06 23:02 UTC)