[HN Gopher] Kangaroos and Training Neural Networks (1994)
       ___________________________________________________________________
        
       Kangaroos and Training Neural Networks (1994)
        
       Author : optimalsolver
       Score  : 10 points
       Date   : 2021-05-17 10:39 UTC (1 days ago)
        
 (HTM) web link (webcache.googleusercontent.com)
 (TXT) w3m dump (webcache.googleusercontent.com)
        
       | jmmcd wrote:
       | > In standard backprop, the most common NN training method, the
       | kangaroo is blind and has to feel around on the ground to make a
       | guess about which way is up. A major problem with standard
       | backprop is that the distance the kangaroo hops is related to the
       | steepness of the terrain. If the kangaroo starts on a gently
       | sloping plain instead of a mountain side, she will take very
       | small hops and make very slow progress. When she finally starts
       | to ascend a mountain, her hops get longer and more dangerous, and
       | she may hop off the mountain altogether. If the kangaroo ever
       | gets near the peak, she may jump back and forth across the peak
       | without ever landing on it.
       | 
       | The first part of this is bad. In backprop we know the gradient
       | just fine. And no analogy has been offered for the aspect that
       | makes backprop distinct from every other algorithm mentioned.
        
         | LeegleechN wrote:
         | The gradient we want is the gradient with respect to the
         | process which generated the dataset. The gradient we get is an
         | estimate based on only a handful of samples from that process
         | at a time. The analogy holds up fine.
        
         | CyberShadow wrote:
         | I understand the fragment you quoted to be accurate, and the
         | problem described is mostly ameliorated by optimizers. But, I
         | don't understand your criticism of the quoted text. Would you
         | mind elaborating?
        
       ___________________________________________________________________
       (page generated 2021-05-18 23:02 UTC)