[HN Gopher] Techniques for Training Large Neural Networks
___________________________________________________________________
Techniques for Training Large Neural Networks
Author : todsacerdoti
Score : 80 points
Date : 2022-06-09 16:01 UTC (6 hours ago)
(HTM) web link (openai.com)
(TXT) w3m dump (openai.com)
| Der_Einzige wrote:
| What if you abandoned gradients/back-prop entirely? They knew
| that for reinforcement learning this was good clear back in 2017!
|
| https://openai.com/blog/evolution-strategies/
|
| Put engineering effort into creating fast, GPU powered
| (tensorflow/pytorch) algorithms for neuroevolution of the
| _weights_ of a neural network. I 'm still not convinced that
| properly leveraged gradient free/Feed-forward only optimizers
| have been tried yet by researchers - namely because they never
| actually wrote fast gradient free optimizers!
| erwincoumans wrote:
| Black box optimization such as CMA-ES or Augmented Random
| Search (ars) are used by researchers, for example:
|
| CMA-ES: https://blog.otoro.net/2017/10/29/visual-evolution-
| strategie...
|
| ARS: https://arxiv.org/abs/1803.07055
|
| Those optimizers are efficient up to certain neural network
| sizes.
| pchiusano wrote:
| You might like these posts on gradient free optimization:
|
| http://pchiusano.github.io/2019-07-06/learning-without-a-gra...
|
| http://pchiusano.github.io/2020-10-12/learning-without-a-gra...
|
| I think you're right that the area is underexplored.
| eismcc wrote:
| Has anyone tried using another NN as the "gradient" function?
| With the hope of making it less "blind?"
| rolisz wrote:
| DeepMind has a thing called synthetic gradients:
| https://www.deepmind.com/blog/decoupled-neural-interfaces-
| us....
| liuliu wrote:
| 1. I missed this paper https://www.microsoft.com/en-
| us/research/uploads/prod/2018/0... when doing my own feature map
| compression research: https://liuliu.me/eyes/reduce-
| another-70-memory-usage-for-de.... Thanks for point it out!
|
| 2. Otherwise, most of these are obvious optimizations. One widely
| popular optimization (that I consider non-obvious) is ZeRO-
| Offload, in particularly the gradient sharding scheme (although
| once learned, the sharding itself is pretty straightforward, just
| a bit chatty). One thing I think undervalued these years though
| is Alex's "One weird trick": https://arxiv.org/abs/1404.5997.
| This scheme is much convoluted but very effective when training
| large MLP. It is not popular probably because a). the
| implementation is non-obvious; b). large MLP fall out of fashion
| quickly, and the computation shape for transformer looks probably
| very different from the MLP its originally trying to solve (with
| 4096 activations per layer).
| mirker wrote:
| Seems similar to the Huggingface tutorial:
| https://huggingface.co/docs/transformers/parallelism.
|
| What's different?
| sydthrowaway wrote:
| Amazing
| superb-owl wrote:
| I'm really excited to see folks starting to talk about
| parallelizing machine learning. The conversation has been
| dominated by GPU-friendly techniques - a classic example of
| "everything looks like a nail when you have a hammer".
|
| I hope we start seeing more massively parallel training
| strategies (most likely with GPUs under the hood still)
| ShamelessC wrote:
| Strange comment. May need to edit something? I can't parse it -
| parallel (e.g. CUDA) techniques have been popular since
| AlexNet.
| Tostino wrote:
| I parsed that as meaning something more like BOINC to
| distribute the work and piece together the final result.
| kristjansson wrote:
| > starting
|
| This is a strange comment to see. The techniques in TFA have
| been at the heart of a the last few years of progress in large-
| scale language models, image generation, etc?
___________________________________________________________________
(page generated 2022-06-09 23:01 UTC)