[HN Gopher] The difficulty of computing stable and accurate neur...
___________________________________________________________________
The difficulty of computing stable and accurate neural networks
Author : programd
Score : 46 points
Date : 2022-03-19 16:41 UTC (2 days ago)
(HTM) web link (www.pnas.org)
(TXT) w3m dump (www.pnas.org)
| cs702 wrote:
| Huh. I can read and understand the abstract and the introduction,
| but I can't judge the work after a first pass. This is the kind
| of paper that _cannot_ be easily skimmed, because it consists
| almost entirely of densely packed pages chock-full of highly
| abstract mathematical reasoning.
|
| Not surprisingly, the authors are mathematicians. They claim to
| _prove_ that
|
| * there are well-conditioned problems for which suitable DNNs
| exist, but no training algorithm can find arbitrarily good
| approximations of those suitable DNNs;
|
| * it's possible to find approximations of those suitable DNNs
| only if we sacrifice digits of accuracy -- i.e., the
| approximations cannot be arbitrarily good; and
|
| * there is a class of DNNs they propose, which they call "fast
| iterative restated networks" or FIRENETs, that solve undetermined
| systems of linear equations over the complex numbers, with a good
| blend of stability (robustness to adversarial samples) and
| accuracy (within the claimed theoretical limits).
|
| Finally, the authors provide open-source code (A+ for doing that,
| but... Matlab!!??): https://www.github.com/Comp-Foundations-and-
| Barriers-of-AI/f...
|
| Does anyone else here understand the work better than me? I would
| love an informal explanation that appeals to intuition.
| lp251 wrote:
| This is a follow-up to an earlier paper on instabilities in NN-
| based solutions to inverse problems:
| https://www.pnas.org/doi/10.1073/pnas.1907377117
|
| Your statements appear to be a good summary of the paper.
| cs702 wrote:
| > Your statements appear to be a good summary of the paper.
|
| Thanks. I found the authors' claims relatively easy to grok.
| What I'd like to understand, intuitively, is how they got
| there!
| lp251 wrote:
| Ah, sorry, I can't help too much there.
|
| I skimmed through the proofs in the SI. Many of the tools
| they use are taken from the compressed sensing literature
| (robust null space property, sqrt LASSO paths, sample
| complexity for CS-MRI style problems...).
|
| I can point you to additional reading, if you'd like.
| glitchc wrote:
| What's wrong with Matlab? It works very well for this, as it's
| very stable for numerical modeling work.
| cs702 wrote:
| It's closed-source; it lacks the kind of
| tooling/infrastructure you need for working with larger,
| state-of-the-art DNNs; and very little AI research is done
| with it, limiting the audience that can can use the code.
| Python -- or even Julia -- would have been a better choice.
| lp251 wrote:
| The story has changed somewhat recently, but up until ~2020
| most ML for Inverse Problems research was done in Matlab,
| especially for MRI and CT reconstruction. Thinking of
| FBPNet and ADMM-Net.
| sjg007 wrote:
| Interesting work. A DNN is after all something that compresses
| the input data and hopefully generalizes well over the domain.
| This 'machine' itself will adhere to principles of Kolmogorov
| complexity etc..
|
| Traditionally, empirical solutions in the NN literature to
| address instability are regularization and dropout.
|
| Also, adding layers seems to improve things. The famous example
| is XOR which cannot be trained by a single layer NN.
|
| How do the theoretical limitations in the paper relate to these?
| if at all...
| kettleballroll wrote:
| PNAS is not exactly the venue of first choice for publishing AI
| research, so heuristically speaking, this article is likely not
| worth the reading time. Furthermore neither the abstract nor the
| first few words of the introduction give me any reason to read
| on. I'm going to assume this is irrelevant, until someone here
| can convince me it's not.
| karpierz wrote:
| They're showing that there are classes of problems that neural
| networks cannot be trained to consistently approximate,
| regardless of the amount of data provided. It sounds like
| you're not interested in the limitations of neural networks, so
| this paper isn't really relevant to you.
| pizza wrote:
| You would take the opinion of a random stranger on HN over the
| article getting past peer reviewed and published in PNAS.. Why
| that heuristic?
| exdsq wrote:
| OP greatly overestimates the knowledge of the average HN
| user!
| kettleballroll wrote:
| I'm a researcher in the field, and the amount of new research
| coming out each day is just way too big to handle. Which
| means I have to rely on heuristics like this to judge what is
| worth my time. Otherwise I'd be spending 25 hours/day reading
| papers, and waste brain capacity digesting worthless
| findings. Not everything that passes peer review is read-
| worthy, far from it. It just means there aren't any gaping
| holes in it. Don't get me wrong, the heuristic is far from
| perfect. But a random paper on HN (which is not known for its
| abundance of researchers) that was posted without any context
| explaining why I should care is not a good signal.
| exdsq wrote:
| If you're a researcher in the field is it not a fundamental
| skill to be able to make these judgement calls yourself?
| And if wasted time is a concern to you, what makes you
| think others would be willing to put in the time to try and
| convince a random academic that they should or shouldn't
| read a paper where they've not even specified _what_ they
| 'd want to get out of it?
| sjg007 wrote:
| I am a researcher in this field and PNAS is fine medium to
| communicate. This paper actually first appeared in arxiv
| which is where everything seems to go first. Having it go
| from arvix to PNAS is a good indicator that it is a good
| paper, and it is! It would be a shame to dismiss it.
|
| Hinton's paper in 2006 on reducing dimensionality with NNs
| appeared in Science and nobody paid attention to that
| either at the time.
| rmellow wrote:
| You might as well say it's not worth interviewing anyone who
| didn't graduate from an Ivy League school, "heuristically
| speaking".
|
| Innovation can come from anywhere. Either we can take time to
| evaluate something on its merits (if you have the competency
| and resources), or leave it be (if you don't), or we accept to
| be told what is true by institutions with their own interests,
| who will use their position to further leverage them. And in
| this path, knowledge is no longer free.
|
| I will read further, but this paper is attacking a real issue
| with neural nets.
| RicoElectrico wrote:
| Still, the choice of journal is pretty weird. Occam's razor:
| If you have any results that feel significant and not just to
| bump your publication count (the title would imply former)
| you'd run with it to present on NIPS, ICLR, ICML or submit to
| JMLR.
| dchftcs wrote:
| It's fair point that it's hard to spend time on something
| if it doesn't carry enough promise. I've seen enough
| crackpots from good universities that don't get called out
| enough. They still exist on the likes of NeurIPS/ICML
| though.
|
| However, to me this appears to be serious work (whether or
| not it's remarkable). Crackpots often obscure their
| message, but here they state their premise quite clearly,
| and skimming through the arguments they seem sane and
| approachable.
| hhmc wrote:
| > You might as well say it's not worth interviewing anyone
| who didn't graduate from an Ivy League school, "heuristically
| speaking".
|
| I know this isn't the point you were going for, but this _is_
| a heuristic that is deployed in some companies/institutions
| (at least the UK variant certainly is).
| exdsq wrote:
| > I know this isn't the point you were going for, but this
| _is_ a heuristic that is deployed in some
| companies/institutions (at least the UK variant certainly
| is).
|
| Which companies/institutions are you thinking of?
| kettleballroll wrote:
| Innovation can come from anywhere, but the authors of this
| paper chose to submit to a venue that is outside the focus of
| ML research. From what I understand, this paper would've been
| a great fit for COLT (or even a general/broad ML conference
| like ICML) and since it wasn't submitted (or wasn't
| accepted), i think it makes sense to assume that it was
| written by authors who either chose to circumvent the
| community, or were rejected by it. I don't find your
| comparison to ivy league fitting, because while there are
| still a significant number of great candidates from non ivy
| leagues, the number of great ML papers that have not been
| published at top tier ML venues is in the single digits --
| most researchers I know would rather just upload their paper
| to arxiv and move on, rather than publishing in 2nd tier
| venues. Which should tell you how much of a bad signal the
| wrong venue can be in ML research.
|
| Granted, it could also be that this was authored by people
| outside of the field who.sont know the rules, but given how
| much research gets published each day, it's impossible not to
| rely on heuristics, especially since this paper was.posted
| without context.
| exdsq wrote:
| If signaling is what gets you off, it might help that the
| authors are based at Cambridge
___________________________________________________________________
(page generated 2022-03-21 23:02 UTC)