[HN Gopher] A new link to an old model could crack the mystery o...
___________________________________________________________________
A new link to an old model could crack the mystery of deep learning
Author : the__prestige
Score : 142 points
Date : 2021-10-12 12:38 UTC (10 hours ago)
(HTM) web link (www.quantamagazine.org)
(TXT) w3m dump (www.quantamagazine.org)
| randcraw wrote:
| The article is intriguing, but is the objective of the
| mathematicians of more than theoretical value?
|
| It seems to me that proving DNNs to be equivalent to SVM kernels
| won't improve them appreciably -- they won't become more accurate
| or precise, not faster, nor more economical. More importantly,
| the achilles heel of DNNs won't be fixed -- their inability to
| explain their reasoning or annotate their inner workings to
| enable tuning or repair.
|
| Replacing black box linear algebra with black box math does not
| seem like much of a step forward.
| JustFinishedBSG wrote:
| > is the objective of the mathematicians of more than
| theoretical value?
|
| Why should it have any other value?
| charleshmartin wrote:
| agreed!
| shenberg wrote:
| Microsoft research has a really nice paper showing that the NTK
| explanation is very unlikely: https://www.microsoft.com/en-
| us/research/blog/three-mysterie...
| YeGoblynQueenne wrote:
| The article leaves "generalisation" without further explanation
| and as usual, there's a lot of confusion about what, exactly, is
| meant by the claim that overparameterised deep neural networks
| "generalize astoundingly well to new data".
|
| So, what is meant is that such networks (let's call them complex
| networks for short) generalise well _to test data_. But what
| exactly is "test data"? Here's how it works.
|
| A team has some data, let's call it D. The team partitions the
| data to a training and test set, T1 and T2. The team trains their
| system on T1 and tests it on T2. To choose the model to test on
| T2, the team may perform cross-validation which further
| partitions T1 to k validation partitions, T11, ..., T1[?]. In the
| most common cross-validation scheme, k-fold cross-validation, the
| team trains on k-1 validation partitions of T1 and tests on one
| of the k validation partitions, trying all combinations of k-1
| training and 1 testing, partitions _of T1_ (because we're still
| in validation, which can be very confusing). At the end of this
| process the team have k models each of which is trained on k-1
| different partitions, and tested on one of k different
| partitions, of the training set T1. The team chooses the model
| with the best accuracy (or whatever the metric they're measuring)
| and then they test this model on T2, the testing partition.
| Finally, the team report the accuracy (etc) of their trained
| model on the testing partition, T2, and basically claim (though
| generally the claim is implicit) that the accuracy of their best
| model on T2 is a more or less accurate estimate of the accuracy
| of the model on unseen data, in the real world, i.e. data not
| included in D, either as T1 or T2. Such truly unseen data (it was
| not available _to the team_ at training time) is sometimes
| referred to as "out of distribution" data and let's call it that
| for simplicity [1].
|
| So, with this background at hand, we now understand that when the
| article above (and neural net researchers) say that complex
| networks "generalise well", they mean "on test data". Which is
| mildly surprising given that various assumptions make it less
| surprising that they'd do well on training data etc etc.
|
| There are two key observations to make here.
|
| One is that complex networks generalise well on test data _when
| the researchers have access to the test data_. When researchers
| have access to test data, the training regime I outline above
| includes a further step: the team looks at the accuracy of their
| system on test data, find it to be abysmal, and, crestfallen,
| abandon their expensive research and start completely from
| scratch, because obviously they wouldn't just try to tweak their
| system to perform better on the test data! That would be
| essentially peeking at the test data and guiding their system to
| perform well on it (e.g. by tuning hyperparameters or by better
| "random" initialisation)!
|
| I'm kidding. _Of course_ a team who finds their system performs
| badly on test data will tweak their system to perform better on
| the test data. They 'll peek. And peek again. Not only they'll
| peek, they'll conduct an automated hyperparameter-tuning search
| (a "grid search") _on the test data_! That is what's known in
| science, I believe, as a "fishing expedition".
|
| Two, in the typical training regime, the training partition T1 is
| 80% of D, the entire dataset, and T2 is 20% of D. And when T1 is
| partitioned for cross-validation, the validation-training
| partition (the k-1 folds trained on) is also usually 4 times
| larger than the validation-testing partition (the single fold
| tested on). Why? Because complex networks need Big Data to train
| on. But, what happens when you train on 80% of your data and test
| on 20% of it? What happens is that your estimates of accuracy are
| not very good estimates, because, even if your data is Really
| Big, 20% of it is likely to miss a big chunk of the variation
| found in the 80% of the data, and so you're basically only
| estimating your system's accuracy on a small sub-set of the
| features that it needs to represent to be said to "generalise
| well". For this reason, complex networks, like all Big Data
| approaches, are pretty much crap at generalising to "O.O.D."
| data, or, more to the point, estimates of their accuracy on OOD
| data are just pretty bad. In practice, deep neural net systems
| that have amazing performance "in the lab", can be expected to
| lose 20-40% of their performance "in the real world" [2].
|
| Bottom line, there's no big mystery why complex networks
| "generalise" so well and there is no need to seek an explanation
| for this in kernel machines. Complex networks generalise so well
| because the kind of generalisation they're so good at is the kind
| of generalisation achieved by humans tweaking the network until
| it overfits _to the test data_. And that's the worst kind of
| overfitting.
|
| __________
|
| [1] Under PAC-Learning assumptions we can expect the performance
| of any machine learning system on data that is truly "out of
| distribution", in the sense that it is drawn from a distribution
| radically different than the distribution of the training data,
| to be really bad, because we assume distributional consistency
| between training and "true" data. But that's a bit of a quibble,
| and "OOD" has established itself as a half-understood jargon term
| so I let this rest.
|
| [2] I had a reference for that somewhere... can't find it. You'll
| have to take my word for it. Would I lie to you?
| YeGoblynQueenne wrote:
| Addendum: to answer the obvious question, it's very difficult
| to test on unseen "OOD" data, because you don't have the
| "ground truth" for it. It is possible to do such "extrinsic"
| evaluation of a machine learning system but it takes time -
| time generally spent with the system operating in a real-world
| environment ("in deployment"). By that time, the system you
| want to test might already be obsolete, because better (lab-
| based) results have been published and there's no point in
| trying to publish an extrinsic evaluation of an old system,
| especially if the extrinsic evaluation shows that performance
| degraded by 20% compared to the lab-based evaluation, let alone
| the new SOTA.
|
| So results on truly-unseen, OOD data are never published, if
| they are ever collected at all, and the field advances one
| meaningless published benchmark-beating SOTA result at a time.
| In the end, despite all the hype, nobody has any idea how well
| neural networks, er, work, in practice. We only know how well
| they do on test datasets.
|
| Oh, alright. Some people who deploy large neural nets for
| google and friends have an _intuition_ about how well they
| work. A ... feeling.
| flerovium wrote:
| Why do I keep seeeing quanta magazine on hackernews? They've
| published a lot of very speculative results, often with simple
| inaccuracies.
|
| Can I filter this out?
| sgt101 wrote:
| By the time I finished my Ph.D. in 1997 I was stone cold certain
| that SVM's were equivalent to NN's, and that 3 layers could do
| anything. I saw Y.Lecuns work on handwriting and thought it an
| oddity. I gave up on machine learning as I thought it was all
| done and concentrated on applications.
|
| In 2011 I was staggered to realise what a load of bollocks I had
| been bongoogled into believing in, and annoyed.
|
| I am never going to fall for the "if we assume x, and y and only
| talk about this bit of the system then it's all the same" line
| again. It may be mathematically nice to prove things with
| simplifying assumptions, but the ICML community basically walked
| up a blind ally because of this desire for rigor and elegance in
| preference to reality.
|
| ML is not mathematics, because it's tied to data, and data is a
| product of nature. The study of nature is Science. This is the
| difference between String Theory and Physics, and it's the
| difference between Computer Science and ML as well.
| jhgb wrote:
| Isn't it still the case mathematically that 3 layers can do
| theoretically anything? Which of course may be a different
| claim that they can do that practically.
| antognini wrote:
| Yes. A sufficiently wide neural network with a single hidden
| layer can approximate any function arbitrarily closely.
| andbberger wrote:
| The missing bit of theory is in explaining the dynamics of
| training deeper networks
| amelius wrote:
| The problem with deep learning is that the functions are
| not fully specified. E.g. you can train the truth table of
| the XOR function. But you can't train on all possible cat
| pictures. The question is: will a single layer even train
| well on a subset of all possible cat pictures?
| YeGoblynQueenne wrote:
| >> But you can't train on all possible cat pictures.
|
| I wanted to make a joke here that, sure you can, all you
| need is an infinite number of cat pictures and infinite
| time and compute, but actually you don't need that.
| Neural nets can do better at identifying cats in images
| than anything that has gone before and that has to be
| said.
|
| The problem is that this better performance degrades
| precipitiously with any distance from the standard
| benchmark datasets that may contain cats. In the real
| world, with its infinite variation of not only cats but
| also cat poses, environments, lighting conditions, etc
| etc, it's a much worse deal and as I say in another
| comment, good luck knowing how well SOTA approaches do at
| it - we'll never know.
| amelius wrote:
| > Isn't it still the case mathematically that 3 layers can do
| theoretically anything?
|
| What do you (and parent) mean by "do"? Inference, or
| training, or both?
| sgt101 wrote:
| The point is that a three layer network can approximate any
| function, and back propagation can be used to train it to
| do so. You could train a 3 layer net to do imagenet, I
| think it was Minsky that showed the first bit of that in
| "Perceptrons", and Hinton showed it in his backprop paper
| with Rumelheart (?) but I can't be arsed to look it up,
| someone will pop up and show that it was actually Leonardo
| Da Vinci or that German fella who has apparently invented
| everything ever and never tires of telling everyone about
| it no matter how hard we all ignore him.
|
| However, clever people (Hinton, Lecun, others v.smart and
| determined and all should get a lot of credit) invented
| some tricks to train networks faster (and carefully showed
| that they really worked, despite what people said about
| them, especially people at ICML), and computers got many
| faster, and it became obvious (which it was to anyone who
| both could do the sums and had the imagination to think
| about it - which I didn't, at all, sad for me) that broad
| networks are much much slower to train than deep networks
| for an equivalent number of neurons (if you use tricks like
| regularization and dropout) and this makes large deep
| networks feasible.
|
| I did play with deep networks in like 1994, because I wrote
| a C program to let me write neural networks - and I could
| do anything I liked with it on my Sun workstation and my
| supervisors had _no idea_ about anything. However, I set
| this "deep" network off training to see what would happen
| and went to the pub. When I came back I interrupted it and
| looked at the model file to figure out how much the
| parameters had changed since initialization (I didn't have
| any of the good small value initialization stuff, I think I
| set them to 0) and they had changed so little that I did
| some sums and decided that the model would not be finished
| training before I finished grad school if I finished at
| all.
|
| So, I went off and built various types of booster on an MPI
| network that most of my colleagues didn't realize was
| running on their nice Sun workstations instead. Talking to
| other people in pubs at conferences has since convinced me
| that most of the grad students working on ML in the 90's
| did the same thing.
| robomartin wrote:
| A while ago I put a stack of early 80's AI books (~1982)
| on my son's desk and said "Have a look through these and
| tell me what you think".
|
| About thirty minutes later he came back and said
| something like: "Shit! That's all the stuff we are doing
| now!".
|
| To which I said: "Yeah. Computers got faster and we have
| lots more storage and memory. Beyond that, I don't think
| we moved forward very much at all."
|
| If you look though 30 to 40 year old literature on AI you
| will find pretty much everything you'd expect in a modern
| book on AI. Sure, yes, there have been advances, of
| course. I would challenge anyone to find 40 years of
| advances when comparing AI then and now.
|
| The best hypothesis I can put on the table is that speed
| and massive storage made a range of applications possible
| over time. With those exhausted, we hit the wall because
| we are essentially working with 30 to 40 year old AI
| technology.
| r-zip wrote:
| I'd recommend anyone interested in the NTK and its limitations to
| check out Greg Yang's work [1] and the recent book by Roberts,
| Yaida, and Hanin [2]. There's clearly a lot more work to be done
| on neural network scaling limits; the NTK is overly simplistic
| (though useful).
|
| 1: https://www.microsoft.com/en-us/research/people/gregyang/
|
| 2: https://arxiv.org/abs/2106.10165
| scythmic_waves wrote:
| Agreed, Greg Yang's work is really interesting.
|
| Here's a reddit thread by Yang discussing his research into why
| NNs are not Kernel Machines [1].
|
| [1]
| https://www.reddit.com/r/MachineLearning/comments/k8h01q/r_w...
| mark_l_watson wrote:
| Is there much mystery?
|
| SGD in a small number of dimensions is easy enough to understand,
| and as Hinton said in an online class I took years ago: to
| visualize very high numbers in dimensions, close your eyes and
| shout out the number. He was joking but his comment convinced me
| to disregard the concept of millions of billions of dimensions
| and instead concentrate on practical techniques that enable
| training many layers, architectures that are multi-headed, etc.,
| and use model architectures that are known to work for different
| types of problems.
| light_hue_1 wrote:
| The big shortcoming of the article is that it doesn't answer
| this question. Why even bother?
|
| > Is there much mystery?
|
| There is much mystery!
|
| Why do some network architectures work better than others even
| with the same number of parameters?
|
| Are all network architectures equivalent? In the sense of, say
| I have a choice of feed-forward convnet vs fully-connected vs
| MLP mixer vs transformer vs whatever else you want. Can I take
| my working transformer and get an equivalent convnet that
| trains the same way? Or is there something special about one or
| more of these architectures?
|
| Are there non-network systems that have the same properties as
| deep networks? Maybe they're better?
|
| What even is the space of non-equivalent models? Right now we
| basically just try stuff and hope to see some improvement.
| That's really unacceptable. If we had theory to tell us if two
| architectures are "the same" (there are many nuances here, just
| as there are when you are say comparing two Turing machines),
| we could talk about the space of all models.
|
| When things work or don't work, is it because of some property
| of our optimization procedure or something inherent in the
| network initialization or architecture?
|
| Why is optimizing some architectures so much easier than
| others? Transformers are great, but they're very unstable and
| tricky to train. As we add layers or modify a transformer, what
| do we do to update our training regime?
|
| How much data do I need to train a network that does X? Can we
| predict this form some property of the task and the network?
| What do I know that I've saturated. If I've saturated what do I
| need to do to get improvements from more data?
|
| Then we get to more serious problems. There are things that
| networks are terrible at, like long-range inferences and
| compositionality. Why is this? We have some fixes to some
| network architectures, but how do we generalize these?
|
| And more broadly. We know basically nothing at all about
| recurrent networks. We thought they were useful, but then we
| essentially gave up on them completely with transformers.
| What's the deal? Are recurrent networks special? Are they
| approximated so well that they don't matter at all?
|
| This all guides us toward a stranger mystery. We know the brain
| has recurrent connections. Actually, the visual system in your
| brain has more feedback connections than feedforward
| connections. Why don't we need to be able to build models that
| have these?
|
| This just starts to scratch the surface about what's mysterious
| here. We're in dire need of some guiding theory, but it remains
| to be seen if this is it. So far theory has contributed nothing
| :(
| r-zip wrote:
| That's a fine attitude for applied ML, but aren't you even a
| little bit interested in why these things work so well? It's
| not obvious.
| datameta wrote:
| It's fascinating how embedding informational complexity in
| higher order relationships somehow gives rise to a map of
| represented concepts that are extendable with new data.
| Explainable NN models are going to be an absolute necessity -
| such as in the justice system or if we employ them in
| autonomous driving.
|
| Tangentially, I don't believe GPT-N will be the way to
| achieve AGI, however we can learn much on said road
| travelled.
| yomama99 wrote:
| > or if we employ them in autonomous driving.
|
| This is not looking to be the case right now.. Companies
| are doing quite well without explainable models. SDC
| companies like Waymo are hoping it will be sufficient to:
|
| 1. Produce a low failure rate estimate, based on millions
| of virtual miles/fake cities.
|
| 2. Combine several NN outputs in a logical way. That way
| there are at least some interpretable values that lead to
| the result. e.g. a separate model that produces candidate
| trajectories of surrounding vehicles, one to perceive
| sensor input, one to plan, etc.
| JustFinishedBSG wrote:
| > Is there much mystery?
|
| Yes ?
|
| > SGD in a small number of dimensions is easy enough to
| understand
|
| Is it?
| qPM9l3XJrF wrote:
| >By all accounts, deep neural networks like VGG have way too many
| parameters and should overfit.
|
| I keep seeing this claim pop up but I've never seen a citation to
| support it. My understanding is that e.g. statistical learning
| theory lets you prove theorems of the form "IF it has this number
| of parameters or fewer, THEN it won't overfit". First year logic
| is enough to know this theorem tells you nothing in the scenario
| where the model has lots of parameters. Can anyone provide a
| solid citation to support the claim that models with many
| parameters "should" overfit?
|
| BTW, another example of a model with lots of parameters which
| doesn't overfit is random forests / gradient boosting. I'm
| wondering if this "should overfit" claim is less of a global
| property of machine learning models in general, and more just a
| property of the particular models that this set of researchers
| have most familiarity with (SVMs).
| JustFinishedBSG wrote:
| Yes usual results in statistical learning theory are upper
| bound on the generalization error, so if you have an
| "overparametrized" (number of parameters is actually a useless
| metric but that's another discussion altogether) then the best
| you can say is "well I expect the error to be at most A
| FREAKING LOT", which is not a very interesting result.
| homerowilson wrote:
| A nice, short, recent paper along these lines (not mentioned in
| the article) is "Every Model Learned by Gradient Descent Is
| Approximately a Kernel Machine": https://arxiv.org/abs/2012.00152
| JustFinishedBSG wrote:
| I find the majority of the comments in this thread pretty
| concerning and quite honestly shocking.
|
| I didn't expect HN to be basically so obscurantist and to act so
| similarly to medieval alchemists (and that's not being fair to
| alchemists).
|
| So apparently theory is useless, all that matters is the "art"
| and empirical results ?
|
| Especially since quite a few of the comments here dismiss this
| work, and related, with a quick "well it's trivial/ well known?"
| and then procede to give their own personal explanation that show
| a clear misunderstanding of the topic.
| mvcalder wrote:
| When I first read about the NTK I was motivated to try an
| experiment to see if the kernel depended on where in the
| parameter space one evaluated it. I was surprised to see in my
| simple example the answer was "no". Here is a write up I did:
|
| https://arxiv.org/pdf/2104.05874.pdf
| charleshmartin wrote:
| Here's an alternative approach, that actually provides real world
| results
|
| https://calculatedcontent.com/2019/12/03/towards-a-new-theor...
|
| Using techniques from statistical mechanics and strongly
| correlated systems, we can compute the average-case-behavior of a
| real world DNN
|
| We believe we can reproduce some of the results of the NTK by
| using a Gaussian Random Matrix. But if we use a more realistic,
| heavy tailed matrix, we get more practical results
|
| A early fork of the theory has been published in JMLR
| https://arxiv.org/abs/1810.01075
|
| and the empirical results in Nature Communications
| https://www.nature.com/articles/s41467-021-24025-8
|
| and we have an open source tool , weightwatcher, which can be
| used in production
|
| pip install weightwatcher
|
| https://github.com/CalculatedContent/WeightWatcher
|
| Please give it a try and let me know if it is useful to you
| thisnormaluser wrote:
| Nature Communications is not Nature. But whatever.
| [deleted]
| chaxor wrote:
| I actually have a preference towards Nature Communications
| over Nature, as Nature Communications actually allows more
| pages for explanation and as such typically has more rigorous
| research.
|
| On the flip side, Nature often has bolder claims with less
| convincing evidence, which often reads similar to pop-sci
| articles.
| charleshmartin wrote:
| That's fine. Here's a few of my online talks:
|
| UC Berkeley / ICSI:
| https://www.youtube.com/watch?v=6Zgul4oygMc
|
| Stanford ICME: https://www.youtube.com/watch?v=PQUItQi-B-I
|
| and a couple of Mikes's
|
| Institute for Pure & Applied Mathematics (IPAM) :
| https://www.youtube.com/watch?v=fmVuNRKsQa8
|
| Physics Informed Machine Learning:
| https://www.youtube.com/watch?v=eXhwLtjtUsI
|
| and our KDD workshop KDD:
| https://dl.acm.org/doi/abs/10.1145/3292500.3332294
|
| many more talks and papers available
|
| And my favorite, a podcast that has featured LeCun himself:
|
| https://blog.rebellionresearch.com/blog/theoretical-
| physicis...
| [deleted]
| light_hue_1 wrote:
| > By all accounts, deep neural networks like VGG have way too
| many parameters and should overfit. But they don't.
|
| This isn't true at all. You can only make this statement relative
| to a dataset. And VGG is almost always trained on ImageNet. A
| standard VGG16 has 138 million parameters. ImageNet has 1 million
| images each is 244x244 (~50k). There are ~431 input data points
| per parameter in VGG16. VGG is not overparameterized for
| ImageNet.
|
| There are plenty of network/dataset pairs they could have given
| as an example, but this isn't it.
| marcinzm wrote:
| VGG also has regularization (L2, dropout, maxpool) if I
| remember correctly so it's sort of odd to use that as an
| example of overfitting when it explicitly tunes against that.
| I'm guessing the epochs and learning rate were also tuned to
| lower overfitting. The same is true to every other popular
| network out there.
| derbOac wrote:
| I have so many questions about this now. For example, how often
| are people _not_ distinguishing between amount of data versus
| number of cases?
|
| Do we really have any sense of how "effective number of
| parameters" scales with network features? This article had me
| wondering if peoples' understanding of these things is backward
| or something.
| light_hue_1 wrote:
| > I have so many questions about this now. For example, how
| often are people not distinguishing between amount of data
| versus number of cases?
|
| To state your question more broadly. Data has structure
| that's why we can learn in the first place. Can we separately
| talk about the structure vs number of examples of the
| different structures in our data? In computer vision this
| would be sort of like talking about how much information is
| within an object class vs between object class.
|
| We don't know. And we don't have the mathematical tools to
| talk about this today.
|
| > Do we really have any sense of how "effective number of
| parameters" scales with network features? This article had me
| wondering if peoples' understanding of these things is
| backward or something.
|
| It's not backward, it's non-existent. We basically try things
| and when we have too many parameters we see that performance
| isn't increasing (or isn't increasing enough to be worth it,
| or with unsupervised methods and massive datasets we just max
| out the size of network we can train).
| PeterisP wrote:
| Wait, shouldn't you measure the information in _output_ data
| (information about classes of samples) to evaluate
| overparameterization, not all the information in various
| features?
|
| I.e. total overfitting can be achieved by ignoring most of the
| input values and using just as many pixels from the top left
| corner as you need to uniquely identify the source image (which
| is far less than the total number of pixels) and map it to an
| arbitrary class, and for a million images that should be
| theoretically possible with million-ish parameters; i.e. no
| matter how many pixels there were in each image, 138 million
| parameters for classification of a million images _is_
| overparameterized from the classic perspective of what
| overparameterization is (some generalization forced simply
| because it 's theoretically impossible to fit each sample
| individually).
| dekhn wrote:
| I remember being introduced to SVMs in the context of machine
| learning in biology before deep neural networks:
| https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2559884/
|
| I left the ML community in disgust at how unsavory these ML
| methods were.
| ryan93 wrote:
| I mean you are gonna have to expand on that. How can you be
| disgusted by some simple math?
| dekhn wrote:
| I dunno why I find the idea of "finding the hyperplane that
| maximally separates classes" so unpleasant compared to the
| approach that makes sense to me "optimize the weights of a
| parameterized function to minimize an objective function". I
| imagine under the hood they're not really different.
| Generally, I think SVMs appear to statisticians while deep
| learning appeals to physicists and probabilicians.
| ppod wrote:
| The explanations in quanta magazine are so good. They are more in
| depth than a usual pop-sci article, but very followable.
| jonnycomputer wrote:
| Incidentally, yesterday I submitted a wordpress blog post about
| some of these issues.
|
| https://news.ycombinator.com/item?id=28817666
|
| https://matloff.wordpress.com/2020/11/11/the-notion-of-doubl...
| erostrate wrote:
| A lot of effort went into these efforts to understand neural
| networks in terms or kernels or SVMs. To the best of my
| knowledge, these efforts have not inspired useful new
| architectures, nor have they made useful experimental predictions
| on real neural nets, nor have they had any significant impact on
| important machine learning benchmarks.
|
| I think some researchers are refusing to accept the idea that
| machine learning is very much an experimental science today, and
| the (very cool) mathematics of kernels, SVMs, empirical risk
| minimization, bayesian statistics, etc. are simply no longer
| useful in the large scale regime.
|
| My prediction is that there will be a useful "deep learning
| theory" in the future, but it will look a lot more like physics
| (such as Kaplan's scaling laws) than early 21st century machine
| learning mathematics/statistics.
| ChrisLomont wrote:
| >I think some researchers are refusing to accept the idea that
| machine learning is very much an experimental science today,
| and the (very cool) mathematics of kernels, SVMs, empirical
| risk minimization, bayesian statistics, etc. are simply no
| longer useful in the large scale regime.
|
| The same statement could have been said of neural networks for
| a decades, but researchers poking at corners eventually found
| methods to turn them into the useful tools they are now. After
| all, those methods you downplay were created since neural
| networks were not useful at that time, whereas many of these
| were.
|
| I'd not poo poo what researchers decide to poke at. Pretty much
| every breakthrough is people poking at the edges of
| understanding. If solutions or steps were straightforward, then
| it would be engineering, not research.
|
| >My prediction is that t
|
| Mine is that neural networks as we understand them now get
| replaced by much more solid methods, based on the principles
| from scientific machine learning, where sophisticated
| differentiable models that are designed to mimic the problem
| space get tuned. After all, even current neural networks are
| heading that direction. Neural networks are simply too
| simplistic to capture lots of the complexity that problems
| demand (hence the current move past them in many domains).
|
| Gluing linear functions together ad-hoc is simply a low level
| approximation to what can be developed using centuries of
| powerful mathematics to make models.
| civilized wrote:
| > My prediction is that there will be a useful "deep learning
| theory" in the future, but it will look a lot more like physics
| (such as Kaplan's scaling laws) than early 21st century machine
| learning mathematics/statistics.
|
| I respectfully but completely disagree.
|
| All the enabling technologies of deep learning come from
| machine learning and statistical learning theory. Stochastic
| gradient descent, regularization, dimension reduction,
| bootstrap, bagging, boosting: these techniques remain the
| fundamental tools in the deep learning toolbox, and a constant
| source of inspiration for the latest innovations.
|
| Physics has done next to nothing for deep learning in
| comparison. It's as marginal as the kernel SVM stuff. Just
| fiddling around with the same old stat mech / network theory /
| power law stuff the Complex Systems types been doing for the
| last few decades.
|
| Stat mech works great for materials because materials are
| relatively simple things and we have relatively simple
| questions about them. We want to know how they respond to
| electricity, magnetism, heat, pressure, etc. When you turn to
| elaborate gadgets for machine translation or image recognition,
| sure, you can ask similar questions, but the questions just
| aren't as interesting. You'll get plenty of plots and
| histograms and power laws out of it, but it's all going to be
| very superficial. It's not going to tell you how the gadget
| works.
|
| All that said, there's no way to be sure where the big advances
| in deep learning theory are going to come from. Your guess
| might be as good as mine.
| curiousgal wrote:
| This is ironic because the exact same thing could have been
| said about Neural Networks back in the 90s. Just because there
| aren't any practical applications now of "mathy ML", it does
| not mean it's a dead end.
| ThereIsNoWorry wrote:
| Neural networks are very non-linear. The blessing and curse of
| non-linearity is its usefulness while escaping what
| mathematical logic inherently is able to model. So yes, I
| strongly assume there will never be a mathematically sound
| theory that predicts deep learning in any meaningful way. Same
| problem with everything non-linear. From two-body to chaotic
| systems, biological interactions to the stock market. All just
| wishful thinking until Mathematicians give up and instead
| continue to play around in their well-defined esoteric spaces.
| ravi-delia wrote:
| I think you conflate several ideas of understanding here. We
| don't have good ways of predicting chaotic systems, but we
| often know a lot about them. Just knowing they're highly
| sensitive would be hard to confirm without math.
| gmadsen wrote:
| you don't seem to have a background in mathematics. A
| huge(and growing) body of work exists for every example you
| gave. Obviously mathematicians are interested in nonlinear
| systems
| goatlover wrote:
| So you're saying Hari Seldon's psychohistory is BS?
| 6gvONxR4sf7o wrote:
| > I think some researchers are refusing to accept the idea that
| machine learning is very much an experimental science today...
|
| This is like criticizing the physicists working on the
| principles underlying steam engines during the days when people
| were making empirical advances in building steam engines, but
| nobody had figured out all the core principles yet.
|
| Of course they understand that it's an empirical science today.
| That's exactly why they're doing the work they're doing.
| bcaine wrote:
| While I sort of agree that machine learning will end up as an
| experimental science, it's way, way too early to say whether
| the theory relating deep learning to kernel methods (e.g.
| Neural Tangent Kernels) will be useful or not.
|
| As an example, just last week a (huge) paper [1] was put on
| arXiv that used these theoretical methods to analyze a bunch of
| common architecture building blocks (skip connections,
| normalization, etc), and then applied their theoretical
| findings to figure out how to train Resnet like models in
| similar training time without these seemingly "required"
| building blocks.
|
| Deep Learning is still in its infancy in many ways, and this
| type of research takes time, slowly building on successive
| results.
|
| [1] https://arxiv.org/abs/2110.01765
| mellavora wrote:
| When you wrote 'huge', I thought you meant huge potential
| impact; I wasn't expecting 172 pages.
|
| team behind the paper is Deepmind/Google. It is probably
| worth a read.
| abetusk wrote:
| I've only skimmed the article, but the idea looks to be "Kernel
| methods" [0] (and showing their equivalence in the infinite
| middle layer width limit?).
|
| If anyone has more insight into the actual idea, I would
| appreciate an explanation.
|
| In the article, there are two advertisements to older articles.
| One is "New Theory Cracks Open the Black Box of Deep Learning"
| from 2017 and the other is "Foundations Built for a General
| Theory of Neural Networks" from 2019.
|
| [0] https://www.quantamagazine.org/a-new-link-to-an-old-model-
| co...
| leecarraher wrote:
| They also have an embedded link to this
| paper(https://arxiv.org/abs/1810.02054 ) regarding the provable
| convergence of neural networks by non-convex gradient descent
| optimization. You are correct, the researchers do no propose an
| alternative to sgd based training (or at least according to the
| article). It did seem like it was going toward an alternative
| training method, perhaps SMO on kernel SVM, but then it pivots
| to describing the theoretical result for infinte kernel svm
| from the linked article.
___________________________________________________________________
(page generated 2021-10-12 23:01 UTC)