[HN Gopher] Google breaks the trillion-parameter ceiling with th...
       ___________________________________________________________________
        
       Google breaks the trillion-parameter ceiling with the Switch
       Transformer
        
       Author : groar
       Score  : 19 points
       Date   : 2021-01-18 14:25 UTC (8 hours ago)
        
 (HTM) web link (arxiv.org)
 (TXT) w3m dump (arxiv.org)
        
       | mensetmanusman wrote:
       | It's an interesting thought experiment to consider models that
       | have more parameters than there are data points being analyzed.
       | 
       | What does that mean?
        
         | rdlecler1 wrote:
         | All networks are like this. In a network of N nodes you have
         | N^2 potential relationships. That's just simple paired
         | relationships. You can still go to higher order groups.
        
         | igorkraw wrote:
         | It basically means you need to learn about double descent:
         | 
         | https://twitter.com/hippopedoid/status/1243229024085835779
         | 
         | and regularize your models well, implicitly or explicitly. Your
         | model might otherwise memorize the data instead of learning
         | features.
        
       ___________________________________________________________________
       (page generated 2021-01-18 23:03 UTC)