[HN Gopher] No More Adam: Learning Rate Scaling at Initializatio...
___________________________________________________________________
No More Adam: Learning Rate Scaling at Initialization Is All You
Need
Author : jinqueeny
Score : 86 points
Date : 2024-12-18 04:49 UTC (18 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| rob_c wrote:
| Interesting take but:
|
| After a reread it's nice to see the optimizer is faster but how
| long is spent in the optimizer and can adamw be tuned for a low
| memory environment given its greedy to try and reduce the impact
| of statistical noise on gradient calculations.
|
| Note that when training on image1k it only becomes comparable to
| adamw after many epoch and infact performs measurably worse for
| most of the training session. (How significant that is, is up to
| debate and model/task/data)
|
| Why not incorporate 2nd order changes into adamw directly?
|
| The lower memory footprint is nice but it's not immediately why
| this is the case. Is the batch size reduced? Model changed? I'll
| be read this after a 2nd coffee and see if it is more obvious...
|
| Still promising if true.
| yobbo wrote:
| I haven't read more than the abstract of this particular paper,
| but it is expected that training behaves differently
| with/without adam.
|
| The problem with adam is that it keeps one more statistic (the
| same size as the model parameters) in memory. It also adds a
| little computation.
|
| The way to deal with it otherwise is to tune the momentum-
| parameters and clip/limit the gradient in various ways.
| eden-u4 wrote:
| Tried the source code on a toy model: adam took 2 epochs to train
| a 10k parameters model, this didn't achieve anything useful in
| 20.
|
| Tweaked a bit the hyper parameters and such, but nothing.
| Probably a bogus implementation?
| dist-epoch wrote:
| Was the toy model a transformer?
|
| Maybe it's just way too small, you wouldn't use Karatsuba
| multiplication to do 3*5.
| eden-u4 wrote:
| that's a wrong simile given that you would get the same end
| result in both cases.
|
| I'm not using a transformer, just a plain Feedforward, Relu
| and dropout for a simple classifier.
|
| I don't know, I can be wrong. I hope and some toy experiment
| shows that even in low case parameters it works fine as well
| as adam.
| cma wrote:
| Did you set them to use the same memory budget? Adam holds more
| state.
|
| They do say it consistently matches or outperforms despite
| simplicity, and I think that statement means at the lower
| budget for their approach, but a fair comparison fk seems if it
| is at least promising would be take advantage of the lower
| memory read to add more params in their version in the
| comparison.
|
| Also the paper says slow initial convergence, under
| limitations:
|
| > More- over, our methods ensure a steady and stable update
| during training, allowing the model to converge better in a
| given task with sufficient training steps. Thus, we might
| observe that the convergence speed is relatively lower than
| Adam's in the early stage of training; as our primary focus is
| to investigate the effectiveness of the SaI approach, we left
| the acceleration of convergence speed in future work.
| johndough wrote:
| I tried it to train a CNN-based CIFAR10 classifier, which
| worked well (only a tiny bit worse than Adam, but the
| difference might go away with hyper parameter tuning), but the
| optimizer totally failed (loss -> infinity) when training a
| U-Net for an image segmentation task. I had to increase eps to
| 1e-4 and decrease lr to 1e-3 so it would not explode, but that
| made it very slow to converge.
|
| My summary is that the memory savings might be great if it
| works, but it does not work everywhere.
| eden-u4 wrote:
| ah, numerical instability in the warmup stage might be the
| issue then?
| jszymborski wrote:
| Yah I mean that's the rub with SGD... you need to spend a
| non-trivial compute budget on hyperparam tuning, which
| sometimes beats Adam.
|
| Adam, on the other hand, generally gets you pretty good
| results without futzing too much with hyper params.
| akos23 wrote:
| More likely a bogus paper, neither their mathematical reasoning
| nor their experiments seem to hold up if you look at them
| closely.
| Der_Einzige wrote:
| A single main conference publication at a top AI conference
| has ROI in the millions for the first author. I watched
| someone in the middle of their undergrad with a single ACL
| _workshop_ publication get a 150K offer starting. It's
| remarkable that anything real at all is published given how
| perverse the incentives are to blatantly make shit up.
| amunozo wrote:
| It's time to stop the "All You Need" titles. This one does not
| even sound good .
| v3ss0n wrote:
| Need to write an article `All you need should consider harmful`
| .
| scotty79 wrote:
| "Considered harmful articles are all you need"
| 0xdeadbeefbabe wrote:
| Goto is all you need.
| cuuupid wrote:
| It's one of the most irritating snowclones because most of the
| time the papers are not presenting some dramatic leap forward
| like attention.
| akos23 wrote:
| I don't find this very convincing, both from a mathematical and
| experimental standpoint.
|
| It seems their method is equivalent to SGD where the learning
| rate of each tensor is scaled by the number of elements in the
| tensor. The supposed "Signal-to-Noise ratio" they use is just
| gSNR=norm(g)/RMS(g-mean(g)), where g is the gradient w.r.t. a
| d-dimensional tensor and the mean is computed across the elements
| of g. For a zero-mean iid random gradient the elementwise
| mean(g)[?]0. A similar argument probably holds for arbitrary, but
| not completely random high-dimensional gradients, mean(g)[?]0. In
| this case gSNR=sqrt(d), which explains why it is constant over
| time and how it varies across the components of the network.
|
| It also seems the optimal value of their hyperparameter sweeps
| occurs at the edge in almost every case, and a granularity of 10x
| for the learning rate and weight decay is too large to make
| direct comparisons anyway.
| spenrose wrote:
| Something we need is no more papers titled " ... All You Need"
| joshdavham wrote:
| Yeah it's way too much of a cliche at this point.
| etiam wrote:
| I think we're more at the point (or beyond) of it being
| deliberately obnoxious as a failed attempt at humor. But
| maybe I'm underestimating just how idolized that original
| paper is.
|
| At any rate, by now I'm erring on the side of not promoting
| or citing them.
| unnah wrote:
| Based on a quick googling, apparently the original paper is
| "One kitchen is all you need" by Sister Eudocia, Isabelle
| DeVerneil, and Jane Hildebrandt, published in Modern
| Hospital vol. 79 issue 3, pages 120-122, 1952.
| https://pubmed.ncbi.nlm.nih.gov/12992940/
| glial wrote:
| Funny, I always assumed it was a Beatles joke.
| brookst wrote:
| "Attention is all you need" is definitely a Beatles joke.
| 0xdeadbeefbabe wrote:
| "All you need is love" can be a recipe for producing offspring.
| I've been wondering about AI parallels.
| fastneutron wrote:
| "All you need" considered harmful.
___________________________________________________________________
(page generated 2024-12-18 23:02 UTC)