[HN Gopher] Beyond automatic differentiation
___________________________________________________________________
Beyond automatic differentiation
Author : shantanu_sharma
Score : 166 points
Date : 2023-04-14 15:37 UTC (7 hours ago)
(HTM) web link (ai.googleblog.com)
(TXT) w3m dump (ai.googleblog.com)
| cs702 wrote:
| Interesting, but I have questions. The OP shows how SafeRate
| improves plain-vanilla Adam, but almost no one uses that anymore.
| The most commonly used optimizer nowadays is probably AdamW. Does
| SafeRate improve AdamW to the same extent, or at all? How does
| using SafeRate compare with Bayesian hyperparameter search? How
| does it compare with rule-of-thumb hyperparameter selection
| methods? Many people seem to get a lot of mileage with AdamW and
| sensible defaults (e.g., a "one cycle" schedule with a max lr of
| 3e-4, momentum peaking at 0.95, and some weight decay)?
| ipunchghosts wrote:
| Lots of people still use Adam and SGD!
| cs702 wrote:
| Haven't seen anyone use it for large-scale (or largish-scale)
| learning in a long while...
| GaggiX wrote:
| It improves Adam in the full-batch setting, which is not
| usually done or desired since minibatch should regularize and
| improve the loss.
| cs702 wrote:
| Ah yes, that too. I forgot to point it out. Thanks!
| DougBTX wrote:
| Batch size can be used for regularisation, but using it for
| that will limit training performance. From the Google
| Research Tuning Playbook:
|
| > The batch size governs the training speed and shouldn't be
| used to directly tune the validation set performance. Often,
| the ideal batch size will be the largest batch size supported
| by the available hardware.
|
| > [...]
|
| > As long as all hyperparameters are well-tuned (especially
| the learning rate and regularization hyperparameters) and the
| number of training steps is sufficient, the same final
| performance should be attainable using any batch size (see
| Shallue et al. 2018).
|
| https://github.com/google-
| research/tuning_playbook#choosing-...
|
| The ideal case is full-batch with tuneable regularisation,
| just the hardware gets expensive.
| modeless wrote:
| The source you cite is talking about "large" batch sizes
| which are still a miniscule fraction of the training set.
| Their (good) advice has no relevance at all in a discussion
| of full-batch training.
|
| Full-batch training is not the ideal case. It's a recipe
| for overfitting. We don't optimize models for their
| training set performance. We need them to generalize to the
| true data, of which our training set may not even be a
| representative sample. SGD works exceptionally well for
| that.
| ogogmad wrote:
| Haven't looked in any depth: How does this cope with functions
| like f(x) = x - x? Interval arithmetic sometimes produces very
| conservative bounds.
|
| If done well, rigorous interval bounds seem like they could be
| really useful.
| kelseyfrog wrote:
| That's great. Interval arithmetic is an underappreciated area of
| study, not because it's particularly difficult - anyone can grasp
| it in short order, but it's rarely the first, second, or third
| tool used for the job when confronted with a problem. Perhaps
| it's just not prestigious enough compared to more sophisticated
| methods.
| 6gvONxR4sf7o wrote:
| How does this generally work in high dimensions? In 1d, an
| interval is a really simple thing. In higher dimensions, it
| generalizes in a bunch of ways, from a box to an arbitrary
| convex region. Given that practical problems tend to have the
| interesting bits constrained around a lower dimensional region,
| I imagine that the wrong representations (e.g. just an axis-
| aligned box) could make the intervals meaningless. Is that the
| case? Is it simple to overcome that (e.g. just align the box
| along the interesting subspace), or in practice is it harder to
| have useful high dimensional "intervals?"
| longemen3000 wrote:
| seems vaguely like McCornick relaxations?
| https://optimization.cbe.cornell.edu/index.php?title=McCormi...
| modeless wrote:
| > We can use a similar idea to take an existing optimizer such as
| Adam and convert it to a hyperparameter-free optimizer that is
| guaranteed to monotonically reduce the loss (in the full-batch
| setting).
|
| I'm not a huge fan of analyses like this that make one huge
| assumption that never holds in practice. Nobody uses full-batch
| gradient descent. It's not possible in practice or even desirable
| in theory. The stochasticity of SGD is important for performance!
| There is no reason to expect results with unrealistic assumptions
| like these to generalize to the cases people actually care about.
|
| > at the cost of a single additional forward pass that increases
| the wall time for each step by a small factor (about 2x in the
| example above).
|
| So it doubles the training time and FLOPS. Then the graph X axis
| should be wall time or FLOPS when comparing the methods, instead
| of steps.
| 6gvONxR4sf7o wrote:
| So what? This isn't some kind of gotcha. They're being explicit
| about the limitation. The built an interesting thing and showed
| that it does interesting things in a toy setting. This gets me
| excited about future work down this path.
| modeless wrote:
| I disagree that the blog post was explicit enough. Tacking
| parenthesized disclaimers onto the end of your sentences is
| not really very explicit. What they're comparing against
| isn't even Adam; calling it Adam at all is misleading. The
| subtitle of the Adam paper is "A Method for Stochastic
| Optimization". It's not designed or intended for full-batch,
| non-stochastic training.
|
| What they should do is lead that paragraph with the fact that
| this is full-batch, note that this is not normal, and explain
| why they chose to do it anyway. And they shouldn't call it
| plain "Adam".
|
| The paper is probably good, and it's fine to test things in
| toy settings, but you need to be upfront with your
| assumptions and limitations.
| popinman322 wrote:
| I also liked the graph that showed only training loss vs Adam.
| Would have been nice to see validation loss as well; are we
| just helping Adam overfit the data?
| GistNoesis wrote:
| In the paper https://arxiv.org/pdf/2212.11429.pdf In point 5.5
| (page 57) Future work :
|
| "
|
| Promising areas of future work include: * Mini-batch
| optimization. Both the theory and experiments in this chapter
| have been limited to full- batch optimization. However, the MM
| paradigm can be extended to mini-batch optimization (e.g.,
| [54]). A universal mini-batch MM optimization algorithm may
| outperform Adam and AdaGrad on certain large-scale machine
| learning problems.
|
| "
|
| The current limits of this algorithm (point 2 page 56) is that
| the reverse mode bounding is not implemented yet (and related
| work didn't obtain tight bounds) so you can't yet apply it
| directly for function with huge number of input parameters, so
| for the time being the main usage is limited to few hyper-
| parameter optimizations.
| fn-mote wrote:
| > So it doubles the training time and FLOPS. Then the graph X
| axis should be wall time or FLOPS when comparing the methods,
| instead of steps.
|
| Thanks for pointing this out.
|
| Inflating your results by tricky measurement methods has been
| around forever, but it can be hard to spot if you're not close
| enough to the field.
|
| Also, unfortunately, the blog post does not contain _anything_
| about the key idea used. (Well, I couldn 't find it.)
|
| Trust regions have been around forever; that's not the new
| idea.
|
| It would be nice if somewhere there was a piece about "this is
| what we do better". Decoding a scientific article to figure out
| how much fluff it is seems like it has an uncertain return.
| lamontcg wrote:
| So could this be used with something like a trust-region SQP
| solver to remove the tweakable trust region parameter?
| ftxbro wrote:
| I'm interested what fdej thinks about this one. arb has recently
| been merged with flint under his maintainership and is getting
| some generic rings wrapper and more interfaces with other
| languages. But maybe its focus is more like algebraic things like
| modular forms and less like tensor libraries for machine learning
| and GPGPU.
|
| https://news.ycombinator.com/user?id=fdej
|
| https://news.ycombinator.com/item?id=26054520
|
| https://arblib.org/
| ogogmad wrote:
| I've just looked at the documentation for eigenvalues and
| eigenvectors in arblib. I was thinking that:
|
| - Eigenvectors are not computable in regions surrounding
| eigenvalue clashes. Therefore you should never compute them.
|
| - Let M be a symmetric matrix being eigen-decomposed. The
| decomposition should be A = P D P^(-1) where only D is
| interval-valued, and _P and P^(-1) are NOT UNDER ANY
| CIRCUMSTANCE INTERVAL-VALUED_ , and A should include M.
| ftxbro wrote:
| I see what you mean, but say you have such a matrix and there
| are two eigenvalues that you can't separate from each other.
| Then wouldn't their corresponding eigenvectors define like a
| 2d plane? Couldn't there still be some interval for that
| plane, like say that the plane is defined by the linear
| combination of two vectors and both vectors have zero on some
| coordinate, say the x coordinate. The interval could still be
| helpful by saying that any combination will also be zero
| there, right?
| ogogmad wrote:
| That makes sense, but what can be done with it?
| ftxbro wrote:
| sorry I'm not sure what you mean
| fdej wrote:
| This looks quite similar like the "Lagrange models" defined by
| Joris van der Hoeven in
| https://hal.science/hal-01188378/document, a version of Taylor
| models where the constant error term is replaced by an interval
| multiple of x^n.
|
| Representing functions locally is a very interesting problem:
| there are lots of possible approaches using intervals, Taylor
| series, Chebyshev series, etc. Big open design space if you ask
| me.
| ogogmad wrote:
| See my comment above about your linear algebra
| implementation.
| sagebird wrote:
| Can you define a hairy version of a function that misleads at
| every point? Ie
|
| Take a well known function: sin(x). Call it G
|
| Now make a function H that makes hairy version of functions. H
| takes a function f, a slope m, and a probability p.
|
| H returns a version of the function where every value is the
| close to f(x), and p percent of the points have a derivative of
| m. The remaining points have the job of undoing the derivative of
| m and jumping back to f(x).
|
| If you are dealing with real numbers then things can get very
| hairy without ever deviating from f(x). You could turn p up to
| 99.999999% and m to a crazy slope.
|
| (I am not a mathematician, I don't know if these musings are
| misguided)
|
| My point is that if you had a hairy function, you wouldn't be
| able to use gradient decent on it, because a derivative takes the
| slope at a point, and these slopes are misleading. However, you
| could use a numerical derivative, that plots the slope between a
| point and another point separated by a non zero distance D. The
| hairiness would not be detected.
| currymj wrote:
| I think for the application here (analogous to autograd on
| neural networks) you know each individual building block of the
| function (a well-defined neural network layer) so can be sure
| such things will not happen.
| [deleted]
| CaptainNegative wrote:
| Taking g(x) = f(x) + epsilon*cos(bx) adds to the derivatives a
| function that bounces between -b*epsilon and b*epsilon while g
| remains within an epsilon envelope of f. Take b large enough to
| fit your needs, e.g. 100/epsilon times an upper bound on |f'|.
|
| Add a scaled version of the Weierstrass function instead if you
| don't want a derivative at all.
| abeppu wrote:
| My main question is whether this is still helpful in an SGD
| context?
|
| I.e. if you have $f = \sum_i^n f_i$ and with every batch you make
| $f_S = \sum_{j\in S} f_j$ for that batch ... being able to take a
| step which definitely causes $f_S$ to decrease doesn't
| necessarily cause $f$ to decrease / the window for $f_S$ may be
| different than the window for $f$?
|
| If it only really makes sense for cases where you can evaluate
| the whole function of interest at every step, which tend to be
| lower-dimensional anyways, then I think an interesting comparison
| would be to quasi-newton methods like LBGFS. I.e. if you're
| involving the taylor-series machinery, when should you use it to
| pick the direction to step (normal QN) vs when should you use it
| to pick only the step size (this work)?
| nyrikki wrote:
| The newton method has the nasty Wada property arise, which is
| often glossed over as being the same as scale invariant noise
| in class but which is better thought of as a topological
| feature rather than scale invariant noise.
|
| It is indecomposable continua and not the imperfect knowledge
| of initial conditions like chaotic systems often invoke.
|
| Boundary conditions becoming indeterminate is a problem with 3
| or more attractors or exit basins.
| bollu wrote:
| Could you provide a link to some more reading about this? i
| do not know what a wada property is, nor why it is
| topological
| nyrikki wrote:
| The Wada property is when you have 3+ disjoint sets that
| share the same boundary set, at least for the complete Wada
| property.
|
| In Newton's fractal, any circle you draw will either have a
| single root or all roots, no matter how small you draw that
| circle. If your initial conditions are near that boundary
| the exit basin you take is indeterminate because that point
| wise boundary point is the boundary of multiple exit basins
| and isn't simply connected.
|
| This differs from other scale invariant features like self
| similar fractal scattering which will have similar
| 'noise'at different scales but is somewhat deterministic if
| treated like noise as an example.
|
| The Wada property arises in several places like Hamilton
| systems, delayed partials and predator prey with food and
| refuge.
|
| If you look for papers on light exiting binary black holes
| there are some good papers.
|
| This page may help, but the Wada property is counter
| intuitive.
|
| https://users.math.yale.edu/public_html/People/frame/Fracta
| l...
___________________________________________________________________
(page generated 2023-04-14 23:01 UTC)