[HN Gopher] Turing Awardees Republished Key Methods and Ideas Wi...
___________________________________________________________________
Turing Awardees Republished Key Methods and Ideas Without Credit
Author : Luc
Score : 89 points
Date : 2023-12-14 15:50 UTC (7 hours ago)
(HTM) web link (people.idsia.ch)
(TXT) w3m dump (people.idsia.ch)
| Imnimo wrote:
| Just from the title I knew exactly which awardees we were talking
| about, and who was going to be doing the talking.
| meowface wrote:
| I guessed it was them but only because they're the only 3 I
| know off-hand.
| hiddencost wrote:
| Schmidhuber really needs to stop. He's been beating this drum for
| decades and he's wrong.
| martopix wrote:
| If a guy becomes famous almost exclusively for having a beef
| with someone else (and this someone else really doesn't need to
| beat any drum in order to be famous), then something is wrong.
| Basically these days this is what Schmidhuber is associated to.
| Which is kind of sad, really.
| logicchains wrote:
| >If a guy becomes famous almost exclusively for having a beef
| with someone else
|
| He was famous well before that for having one of the best
| labs in Europe, and many key papers in early deep learning.
| He's only famous for the beef among people who aren't
| familiar with his earlier work and the huge contributions it
| made to the field.
| caddemon wrote:
| It's hard not to think of the beef when you think of him
| now though, even if you already knew some of his work. To
| be clear I don't think that should take away from what he
| did accomplish, but it's like a Pavlovian association at
| this point to also have his behavior come to mind.
| logicchains wrote:
| So he should just give up and let people get away with
| plagiarism? Do you teach your kid to just give in and let
| bullies get their way?
| gexaha wrote:
| I wonder what Schmidhuber colleagues think of all of this.
| gillesjacobs wrote:
| Within the field his name has become a punchline to joke about
| self-aggrandizing.
|
| But yeah, you honestly have to wonder what it is like day-to-
| day working with someone with this level of delusions of
| grandeur.
| logicchains wrote:
| >But yeah, you honestly have to wonder what it is like day-
| to-day working with someone with this level of delusions of
| grandeur.
|
| It's not delusions of grandeur; anyone with half a brain who
| read his early papers would see he clearly came up with the
| idea of a GAN well before Ian Goodfellow.
| RcouF1uZ4gsC wrote:
| Reading the article and some of the links make me feel like the
| author Jurgen Schmidhuber is the academic version of the patent
| troll.
|
| It sounds like he published some theoretical musings back in
| 1990s without any real practical implementation that did anything
| useful and since then has run around accusing AI researchers who
| actually produced concrete research and techniques to get are
| actually in use today of plagiarism.
| aborsy wrote:
| Regardless, it should be easy to verify. He cites a paper and
| says it's done in this earlier paper.
|
| On the other hand, if his claims are verified, some of these
| research "discoveries" are simply common sense and would occur
| to most people working on the subject. HLB were awarded to a
| good extent because they worked on deep learning at the right
| time. Deep learning became hugely practical, and outperformed
| the state of the art in many applications. HL also worked for
| major companies.
| logicchains wrote:
| >It sounds like he published some theoretical musings back in
| 1990s without any real practical implementation
|
| He published working models back then, the problem was compute
| power was very limited. In the past decade deep learning took
| off, people took his models, renamed then and ran them on
| vastly more powerful computers, to great success, then failed
| to cite him.
| vbarrielle wrote:
| It's actually a common problem in lots of computing related
| research communities. Papers older than 10 years ago,
| sometimes even 5 years ago, are ignored. Because the results
| do not compare in terms of computing power, and because
| reimplementing old papers is often very hard because they are
| often too vague.
| harveywi wrote:
| LeCuna (noun): An empty space in a list of citations where the
| works of Jurgen Schmidhuber should appear.
| SeanLuke wrote:
| We published a paper a while back in a genetic programming
| conference, and before the paper had been published (only the
| title was announced) Schmidhuber guessed that we had not cited
| his work. He very publically put us through the wringer for not
| citing him as the inventor of the concept were were examining. In
| fact we had _not_ cited him: but among our dozen or so previous
| examples, we had cited two _others_ which long predated his work
| and were the actual seminal papers. He had failed to cite them
| himself.
| tines wrote:
| > He very publically put us through the wringer for not citing
| him as the inventor of the concept were were examining
|
| He put you through the wringer for not citing him in the
| _announcement of the title_?
| calf wrote:
| But it is possible that he ought to have been cited as well? It
| doesn't matter if he's a hypocrite. The justification
| ultimately has to be, "Our paper did not use your work in any
| substantial way therefore a citation is not needed."
|
| I can see how citations can get very complex for the purposes
| of scientific credit, traditionally there's a self-policed
| aspect to it in every academic community; however, in this
| century as advanced research gets more complex and globalized,
| it might be time for a more rigorous and neutral process of
| doing so.
| lacker wrote:
| Whenever you cite papers just because you're "politically
| supposed to", even though you don't think it's really
| intellectually useful for anyone to be aware of the cited
| paper, you're making the citations less useful for everybody.
|
| Schmidhuber is only the worst of many abusers of the citation
| system. Everyone knows that anonymous reviewers are more
| likely to approve of people who cite their work. Someone is
| politically powerful? Better cite their paper, even though
| you don't really think it's a good paper and you didn't use
| it.
|
| Schmidhuber is cited far more often than he should be,
| because people just don't want to deal with this crap, and
| the cost of sticking in one more citation is very low.
| pavel_lishin wrote:
| Cue the next HBomberguy video, please.
| erostrate wrote:
| List of "famous" ML people not to waste time on:
|
| - Gary Marcus
|
| - Juergen Schmidhuber
|
| - Pedro Domingos
|
| - Max Tegmark
|
| - Eliezer Yudkowsky
|
| Some context for people unfamiliar with ML research: the author,
| Schmidhuber, is well known for claiming that he should get credit
| for many ML ideas. Most ML researchers think that:
|
| - He doesn't deserve the credit he claims, in most if not all
| cases.
|
| - There's a few cases where his papers should have been cited and
| weren't. That's fairly common.
|
| - People do not get much credit for formulating an abstract idea
| in a paper or implementing it on a toy problem. Credit belongs to
| whoever actually makes it work.
|
| - Credit assignment in ML is not perfect but roughly works.
| rcbdev wrote:
| While I generally agree with you in these specific cases, I
| feel like this is a shortsighted form of argument nonetheless.
| Coming up with an idea begets a scalable implementation of it
| and it's not very wise to ascribe absolute value solely on one
| side of the equation.
| TimPC wrote:
| I think it's less about scalability and more about
| identifying specific choices that are promising. A lot of
| Schmidthuber's work paints out broad ideas in grand strokes
| and suggests hundreds of potential neural networks without
| evaluating what choices in that massive space are good. He
| then claims credit when other people identify the specific
| one or two of those hundred models (often with minor
| variations that make it not immediately obvious whether it
| perfectly fits the broad definition or not) that are actually
| promising.
| calf wrote:
| But if other people identify and further develop one or two
| out of his models, he should still be cited. If they did
| not use his work at all, coming ith one or two models
| independently, then that's a different situation. It's a
| bit of an honor code thing as well, it's hard to prove if
| somebody has a read a paper or not. But then there's a more
| stringent standard where one cites as part of surveying
| preexisting work, which can result in a vast list.
| mahe_ wrote:
| > Credit belongs to whoever actually makes it work.
|
| This is just plain wrong. No working version without the idea.
| ben_w wrote:
| Ideas are dime-a-dozen.
|
| I independently arrived at[0] something close to Max
| Tegmark's idea of the Mathematical Universe, almost nobody
| noticed and fewer still cared because I published it as a
| LiveJournal blog post whereas he fleshed it out into a whole
| book.
|
| I didn't get credit because I didn't do the hard work that
| deserves credit, I had the flash of inspiration and stopped
| after a few paragraphs of mediocre student philosophy.
|
| [0] and possibly predated, but I lost track of the date
| format when shifting from LJ to WP: https://kitsunesoftware.w
| ordpress.com/2018/08/26/mathematica...
| GlenTheMachine wrote:
| Why Tegmark?
| erostrate wrote:
| He talks a lot about the singularity but the biggest
| singularity I can see is in his ratio of "talks about AI"
| divided by "actual AI contributions".
| senderista wrote:
| It's not his field, what do you expect?
| GlenTheMachine wrote:
| There's nothing wrong with being a communicator to the
| public.
| erostrate wrote:
| Agreed, although I personally prefer when actual experts
| are communicating to the public. But that's not all he's
| doing, Tegmark also intends to influence global policy on
| AI research, random example:
| https://www.theguardian.com/technology/2023/sep/21/ai-
| focuse...
| YeGoblynQueenne wrote:
| >> Credit belongs to whoever actually makes it work.
|
| That is according to whom? Is it a rule you just came up with
| or accepted practice? And if it's accepted practice, in what
| community is it accepted practice? Because where I publish and
| review there's really no such rule and credit belongs to the
| people who deserve credit for the work they've done that was
| useful to others.
| lowbloodsugar wrote:
| Basically ideas are a dime a dozen. Sure, _your_ idea might
| be a good one, but how do we spot your grain of sand is
| special when it looks the same as the rest of the desert?
| Essentially _having an idea_ isn 't useful to others.
| Demonstrating that your idea has legs is useful to others.
|
| I don't have to deal with citing papers, but I once had to
| deal with people pitching me ideas, wanting me to sign an
| NDA, in exchange for 50% of the revenue after I did all the
| actual work. Just out of curiosity, I signed one once. It was
| a fart app, IIRC. They thought a fart app needed an NDA, and
| that I'd then go do all the work and give them 50% because
| they "had the idea". It was so laughably sad.
|
| If you think these ideas are valuable, I have a beautiful
| clock for you. It is right twice a day. You'll have the same
| problem: you won't know when it's right. You'll need someone
| else's work to tell that.
| shrimpx wrote:
| > Basically ideas are a dime a dozen.
|
| There's a spectrum of ideas, from groundbreaking to "dime a
| dozen". In tech startups, and in almost all of computer
| science, most ideas are a dime a dozen, and the value is in
| the execution.
|
| But clearly, some ideas are groundbreaking. Einstein
| rightfully gets the credit for an on-paper hypothesis that
| wasn't proved until decades later via a chain of critical
| discoveries and experimental innovations by other people.
| It's legit to call it Einstin's relativity, and not
| Mossbauer/Hay's relativity.
| erostrate wrote:
| > credit belongs to the people who deserve credit for the
| work they've done that was useful to others
|
| Certainly agree. The point is that coming up with the idea,
| writing it as an equation, or an architecture diagram in a
| paper, is a small fraction of the effort that goes into
| making the idea work in a model showing good performance on
| real life datasets.
|
| For example, just taking a random paper that Schmidhuber
| claims should give him credit for GANs,
| https://people.idsia.ch/~juergen/FKI-126-90ocr.pdf hopefully
| you can easily see that a lot of work would be needed to turn
| this into a realistic image generation model. And that is,
| even if you admit that the idea is strongly related to GANs,
| which I'm not convinced of but won't spend time on.
|
| > Credit belongs to whoever actually makes it work. >> That
| is according to whom? Is it a rule you just came up with or
| accepted practice? And if it's accepted practice, in what
| community is it accepted practice?
|
| It is accepted practice in the ML community. If it weren't,
| Schmidhuber wouldn't be complaining.
| abdullahkhalids wrote:
| In many fields, this is how citations would work
|
| "Introductory theoretical work in GAN was done by
| Schmidhuber [1], but it was not until large experimental
| efforts [2,3,4] on image generations that the power of GANs
| was revealed."
| caddemon wrote:
| I agree a significant amount of work (and often insight
| too) is needed to translate an architecture idea into
| something that works in practice, and there are certainly
| plenty of ideas that are obvious in the abstract. But I
| also think it's important to avoid dismissing work only on
| the basis that it doesn't involve "real life datasets".
|
| Deep learning is a relatively unexplored field and there
| are many open mathematical and scientific questions to ask
| that involve only model equations or contrived datasets.
| Novel theoretical results are not just about some
| architecture idea but about proving facts that can be
| useful for understanding how the model class would perform
| in different scenarios. Which in turn can help shape the
| search space for applied work.
|
| Additionally, I don't think credit assignment should be so
| discrete. 100% agree that vomiting out vague ideas
| shouldn't grant claims to credit, but academic science much
| too often gives only a single author the "real" credit.
|
| Incidentally, in other fields the person who actually makes
| it work very well may not be the person that receives this
| credit. Like biology can involve a lot of hard manual work
| (that isn't really intellectual) in order to realize a
| project plan. It varies how much of the credit those people
| receive, and I'm not even sure how much they should
| receive. This topic is extremely nuanced.
| anonymousDan wrote:
| I don't buy that this is standard practice in the ML
| community, and even if it is it's BS. If the basic
| idea/principle has been published previously but in a
| different context you should cite it and say why the
| solution is not directly applicable or has not been
| evaluated in the current context. Anything else is
| unprofessional.
| screye wrote:
| The sad part is that Schmidhuber is not a grifter by any means.
| If the turning prize for deep learning could go to 5 people
| instead of 3, he would very likely be on that list.
|
| His lab is excellent and was easily Europe's best deep learning
| lab for decades before it blew up.
|
| Some of his complaints are valid too. European labs often get
| ignored, and he has been sidelined despite being one of the
| most important people in deep learning himself.
|
| But man doesn't know when an argument runs out of gas. His
| claims get grander with every passing year.
|
| He would've just been the 'get off my lawn' grandpa of deep
| learning, but he somehow comes across as even more insufferable
| than that.
|
| I wonder if 2023 schmidhuber was created because the polite one
| from a decade ago was ignored. A sort of evil phase, if you
| will.
|
| I feel bad for him. He did get passed over of some deserved
| awards and recognition. But he reeks of resentment and thats
| never a good look.
| throwup238 wrote:
| _> I feel bad for him. He did get passed over of some
| deserved awards and recognition. But he reeks of resentment
| and thats never a good look._
|
| It's a terrible look. We're in the middle of one of the
| biggest gold rushes in tech history and he's wasting time
| complaining when he claims to be one of its pioneers? That
| effort is much better invested in building stuff but I
| suspect he's fallen into the classic PI trap of writing
| grants all the time and leaving the real work to the rest of
| the faculty, atrophying his skills too much to do anything
| now that the industry is moving so quikcly.
| logicchains wrote:
| >atrophying his skills too much to do anything now that the
| industry is moving so quikcly.
|
| His recent papers are still cutting edge. He's already
| solved self-improving AI: https://arxiv.org/abs/2202.05780
| , it just needs to be scaled up.
| KRAKRISMOTT wrote:
| His students you mean, otherwise he would be first
| author. Academia is just like capitalism, with credits
| substituting for capital. The capital owner (laboratory
| head/professor) always gets a cut of everything
| published.
| jltsiren wrote:
| Authorship conventions vary a lot within the academia. In
| general, the closer the name is to either end of the
| author list, the more significant their contributions
| likely were.
|
| Things also vary from paper to paper. Sometimes the first
| author just did the actual work for somebody else, and
| sometimes they also made significant intellectual
| contributions. (If the first author is listed as the sole
| corresponding author, it usually indicates the latter.)
| Sometimes the last/senior author just brought the money
| in, sometimes they were primarily mentoring the first
| author, and sometimes they were the driving force behind
| the project.
| screye wrote:
| Yep, in ML, the head of the lab is always last author....
| even if it was equal contribution work with their own PhD
| student.
|
| The PhD student needs the street cred a lot more than a
| tenured PI.
| logicchains wrote:
| >Most ML researchers think that: He doesn't deserve the credit
| he claims, in most if not all cases.
|
| That deserves a source. Especially for "all cases"; I don't
| think anyone who understands machine learning could read some
| of his earlier papers and still think Ian Goodfellow invented
| GANs.
| refulgentis wrote:
| Frankly, I do, and it comes across as quibbling about
| categories and trying to define unnecessarily general
| taxonomic categories, as a reaction to positive reactions to
| other people.
|
| ex. in the article: "Goodfellow eventually admitted that my
| PM is adversarial...but emphasized that it's not generative.
| However, [it] is both adversarial and generative (its
| generator contains probabilistic units)...It is actually a
| generalized version of GANs."
|
| When you're at "actually, probabilities means generative, and
| actually you know what, even my initial claim was too
| specific: turns out its a generalized version of GANs", all
| in service of arguing a paper should have been cited in
| another paper, years after the other paper has been
| published, there's not much room for sympathy.
| P-NP wrote:
| This sounds a bit like a justification of plagiarism. In
| science, you must cite the original work.
| erostrate wrote:
| Agreed, Ian Goodfellow wasn't the first to come up with the
| idea of jointly training a generator and discriminator. But
| he was the first to make it work for image generation with
| modern neural networks. For that Ian Goodfellow deserved to
| get most of the credit, and he did.
| 1024core wrote:
| Life would be so boring without Schmidhuber.
| gedy wrote:
| As an outsider to this space, it seems suspect that all of the
| people he has a beef with (like LaCun) will gladly cite others
| and predecessors, but not Schmidhuber.
|
| Has he explained his reasons for thinking there is some
| conspiracy? Otherwise it reflects badly on his assessment of
| himself, or possibly his mental state.
| daveguy wrote:
| It's nice of Schmidhuber to point out the quality papers with
| theoretical advancements and actual validations that fix the
| intellectual problems with his weak musings.
| YeGoblynQueenne wrote:
| This whole comment section is full of absolutely unacceptable ad-
| hominem attacks on Schmidhuber, from people who most likely
| haven't even read any of the works in question and are certainly
| not showing any of the "intellectual curiosity" this site is
| supposed to be about.
|
| Anyone who cares about academic integrity should at least not
| attack someone complaining of plagiarism. That sort of attack is
| the academic equivalent of blaming the victim. If even half of
| Schmidhuber's accusations have a basis that's still a major
| academic scandal of epic proportions.
| peterfirefly wrote:
| I tried reading one of his papers that he claimed contained an
| important concept avant la lettre. It was unreadable and it
| required an almost Marxist torture of the text to make it
| confess (weakly and unconvincingly) that it contained anything
| remotely like the idea he claimed it did.
| logicchains wrote:
| >It was unreadable
|
| Which paper was that; I've always found his writing
| relatively clear. Are you sure it wasn't just the case that
| you didn't understand the paper?
| matkoniecz wrote:
| > Anyone who cares about academic integrity should at least not
| attack someone complaining of plagiarism
|
| What about cases when accusation was unreasonable and not
| matching reality?
| dr_dshiv wrote:
| I came to Schmidhuber's work learning about his approach on
| creativity and curiosity and I loved it so much. It's still so
| relevant. And beautiful, actually.
|
| https://arxiv.org/pdf/0709.0674.pdf
| qzw wrote:
| Shouldn't the title be "Turing Awardees Accused of
| Republishing..." rather than how it currently reads?
| yorwba wrote:
| No, because the article author is the one making the
| accusations.
| 33a wrote:
| Somehow I knew it was Schmidhuber before I clicked the link.
| renecito wrote:
| Shocking! Successful people getting credited for someone else
| work.
| cs702 wrote:
| Copying and pasting Urban Dictionary's definitions of "to
| schmidhuber"[a], "schmidhuber"[b] and "schmidhubered"[c]:
|
| ---
|
| to schmidhuber
|
| When you publicly claim that someone else's idea that is remotely
| resembling your own is stolen from you.
|
| ---
|
| schmidhuber
|
| 1) To interject for a moment and explain how one's recent popular
| idea is a few transformations away from your 1991 paper.
|
| 2) To miraculously produce fifty years of relevant literature
| after someone claims to trace the origin of an idea in a
| particular work.
|
| ---
|
| schmidhubered
|
| Being "schmidhubered" looks something like this:
|
| 1) Invent something brilliant that no one cares about. Experience
| derision.
|
| 2) That thing becomes popular years later. Someone else is given
| credit for inventing it. That person appears in the New York
| Times and is declared smartest person alive.
|
| 3) Go on a campaign explaining the situation and how you are the
| rightful inventor and thus the rightful Smartest Person Alive.
|
| 4) Everyone accuses you of being a sore loser and no one takes
| you seriously.
|
| 5) A verb is named after you.
|
| ---
|
| [a]
| https://www.urbandictionary.com/define.php?term=to+schmidhub...
|
| [b] https://www.urbandictionary.com/define.php?term=schmidhuber
|
| [c]
| https://www.urbandictionary.com/define.php?term=schmidhubere...
| P-NP wrote:
| I can see some angry comments here, but so far I have not seen
| any facts that refute his claims. Once I spent a long time
| reviewing a related paper on Hacker News, and I think he is right
| about disputes B1, B2, B5, H2, H4, H5. I'd have to study the
| others more closely:
|
| B: Priority disputes with Dr. Bengio (original date v Bengio's
| date): B1: Generative adversarial networks or GANs (1990 v 2014)
| B2: Vanishing gradient problem (1991 v 1994) B3: Metalearning
| (1987 v 1991) B4: Learning soft attention (1991-93 v 2014) for
| Transformers etc. B5: Gated recurrent units (2000 v 2014) B6:
| Auto-regressive neural nets for density estimation (1995 v 1999)
| B7: Time scale hierarchy in neural nets (1991 v 1995)
|
| H: Priority disputes with Dr. Hinton (original date v Hinton's
| date): H1: Unsupervised/self-supervised pre-training for deep
| learning (1991 v 2006) H2: Distilling one neural net into another
| neural net (1991 v 2015) H3: Learning sequential attention with
| neural nets (1990 v 2010) H4: NNs program NNs: fast weight
| programmers (1991 v 2016) and linear Transformers H5: Speech
| recognition through deep learning (2007 v 2012) H6: Biologically
| plausible forward-only deep learning (1989, 1990, 2021 v 2022)
|
| L: Priority disputes with Dr. LeCun (original date v LeCun's
| date): L1: Differentiable architectures / intrinsic motivation
| (1990 v 2022) L2: Multiple levels of abstraction and time scales
| (1990-91 v 2022) L3: Informative yet predictable representations
| (1997 v 2022) L4: Learning to act largely by observation (2015 v
| 2022)
| senderista wrote:
| This guy has been the biggest blowhard in AI for years if not
| decades.
___________________________________________________________________
(page generated 2023-12-14 23:01 UTC)