[HN Gopher] Foundations Built for a General Theory of Neural Net...
___________________________________________________________________
Foundations Built for a General Theory of Neural Networks (2019)
Author : ColinWright
Score : 50 points
Date : 2021-01-23 12:27 UTC (10 hours ago)
(HTM) web link (www.quantamagazine.org)
(TXT) w3m dump (www.quantamagazine.org)
| dang wrote:
| Discussed at the time:
| https://news.ycombinator.com/item?id=19069671
| orange_tee wrote:
| This is something I am interested and considering as a possible
| future for my career, coming from a math background and starting
| to dip my toes into ML. One thing I am afraid of is how useful is
| it actually going to be to create a rigorous mathematical
| framework? I am afraid it might end up like mathematical physics,
| where they are almost a century behind theoretical and
| experimental physics and playing catch-up. What does anyone else
| think?
| dr_dshiv wrote:
| Relate it to free energy minimization, that's very hot right
| now. Read Smolensky's 1986 article in PDP on the harmonium, it
| was the first restricted boltzmann machine.
| canjobear wrote:
| Don't most neural networks already do free energy
| minimization? Just about any information theoretic objective
| can be interpreted that way...
| dr_dshiv wrote:
| Yes, that's arguably the dominant paradigm. It's
| interesting because of the relationship to thermodynamics
| -- again, check out Smolensky's paper in PDP, he was a
| postdoc (along with Geoff Hinton) at UCSD's cognitive
| science dept, run by Don Norman.
|
| https://apps.dtic.mil/sti/pdfs/ADA620727.pdf
| GregarianChild wrote:
| In what sense are neural nets _not_ rigorous?
|
| The whole pipeline from Pytorch or TensorFlow and Python to
| LLVM, to GPUs or TPUs is _absolutely_ rigorous. Much more
| rigorous, in fact, than normal, hand-written mathematics, as
| you find it in e.g. a typical _Annals of Mathematics_
| publication, or mathematical textbook!
|
| I think what you really have in mind is a simple model of
| modern deep learning that is not fully accurate, but still
| useful!
|
| Let me argue by analogy. You are looking for something that is
| to deep learning what the lambda-calculus is to the Haskell
| compiler. One of the main simplifications in programming
| language theory is replacing finite precision arithmetic (which
| is painfully complex) with mathematical integers and real
| numbers (which are much simpler). Would a theory of deep
| learning based on mathematical reals be valuable in a theory of
| deep learning? The stunning success of floating point formats
| like bfloat16 [1] suggests otherwise, since arithmetic
| precision in deep learning is closely connected to important
| learning phenomena such as overfitting and regularisation.
|
| I am tempted to be provocative and say that you are really
| looking for _less rigour_!
|
| [1] https://en.wikipedia.org/wiki/Bfloat16_floating-
| point_format
| avrionov wrote:
| Not the OP, but I've built several ML pipelines in the last
| few years. Even if every step in the process very rigorous
| and using solid software, there are still challenges. ML
| models are very difficult (or impossible to test). The usual
| testing approaches don't work at all for ML features.
| b3kart wrote:
| We don't know which model architecture will work for which
| problem, and why it would/wouldn't. We experiment until we
| find something that works, and can sometimes try to guess why
| it did. But none of this knowledge is _formalized_ in a way
| that can reliably _predict_ performance in future problems.
| We engineer solutions to problems, but don 't build a
| _rigorous_ body of knowledge to help us in future problems.
| GregarianChild wrote:
| The problem you describe (model architecture will work for
| which problem) is not lacking rigour, but due to the Turing
| complete expressive power of neural networks.
|
| What you can do is use the existing rigour and derive
| stronger properties for _restricted classes_ of NNs (just
| like you can prove stronger properties for simple
| subsystems of lambda-calculus) and that is an interesting
| field of study.
| glial wrote:
| Well put. I would say that often even the problems aren't
| particularly well defined.
| [deleted]
| karpierz wrote:
| > In what sense are neural nets not rigorous?
|
| In the sense that for non-trivial applications, we struggle
| to define what approach is successful and why it's successful
| without leaning on empirical metrics.
| GregarianChild wrote:
| That's not because of lacking rigour, but because of
| complexity!
|
| You cannot, in general, predict arbitrary program
| properties without running programs, that's the essence of
| Rice's theorem [1]. By the famous _Universal Approximation
| Theorems_ for neural networks, in general, NNs are Turing
| complete, which precludes general and simple mathematical
| "silver bullets" that will help you overcome those
| challenges.
|
| [1] https://en.wikipedia.org/wiki/Rice%27s_theorem
| enriquto wrote:
| > This is something I am interested and considering as a
| possible future for my career, coming from a math background
| and starting to dip my toes into ML.
|
| This is a great time and place to be! Neural networks are the
| twenty-first century Fourier series. It's just that we don't
| yet understand them, we can easily run them (synthesis) but we
| are missing the analysis. There's a lot of math to do here.
| joe_the_user wrote:
| One thing to consider is the distinction between deep neural
| networks as mathematical objects and machine learning as
| currently practiced.
|
| Lately, there have been quite a few theories on neural networks
| as ideal nonlinear approximators [1]. Similarly, people have
| shown many ways that gradient descent can tend to reach a
| global maximum of regularized curve-closeness[2]. Which is to
| say, if your development cycle is: gather-data, train, test,
| deploy, we know this approximates the data almost ideally; you
| can't really do much better than a deep network.
|
| But we know in practice, when deployed, that deep neural
| networks actually have many limitations (compared to our
| intuitions or compared to human performance, etc). There are
| some obvious explanations. Of course, they're limited by our
| ability to gather data and by the biases of the data. But even
| more, they're limited to situations where you have large chunks
| of unchanging data that you can extrapolate from.
|
| Given that deeps are more or less perfect for the train-test-
| deploy cycle, it seems like the problem is with this cycle
| itself. And it's easy to see human beings somehow acting
| "intelligently" without using this cycle. So figuring out an
| alternative to this might be something to look at.
|
| [1] For example: Nonlinear Approximation and (Deep) ReLU
| Networks I. Daubechies, et al https://arxiv.org/abs/1905.02199
|
| [2] For example: Gradient descent optimizes over-parameterized
| deep ReLU networks Difan Zou et al
| https://link.springer.com/article/10.1007/s10994-019-05839-6
| bigdict wrote:
| > This is something I am interested and considering as a
| possible future for my career, coming from a math background
| and starting to dip my toes into ML.
|
| I'm in the exact same position.
|
| I think mathematical explanations and (substantiated) intuition
| for the things that practitioners discover would be very
| useful. Maybe an all explaining grand theory of deep learning
| is possible.
| konjin wrote:
| You're 5 years too late to try and break into ML.
| orange3xchicken wrote:
| imo, instead of developing new theory entirely from the ground
| up, its more useful to address & work in the context of
| understanding longstanding open problems/phenomenon.
| Theoretical insight and the framework should follow.
|
| e.g.
|
| Robustness & expressiveness
|
| Memorization & catastrophic forgetting
|
| Ensembling & ntk & optimization
|
| Double descent
|
| Pruning
___________________________________________________________________
(page generated 2021-01-23 23:01 UTC)