[HN Gopher] Enigma: GPT-2 trained on 10K Nature Papers: Can you ...
___________________________________________________________________
Enigma: GPT-2 trained on 10K Nature Papers: Can you spot the
difference?
Author : MrUssek
Score : 116 points
Date : 2021-05-13 17:26 UTC (5 hours ago)
(HTM) web link (stefanzukin.com)
(TXT) w3m dump (stefanzukin.com)
| arthurofcharn wrote:
| Could we feed gpt-2 Turbo Encabulator? I want more Turbo
| Encabulator.
| thereddaikon wrote:
| GPT-2 is Open Source. Nothing stopping you from training it on
| techno babble. GPT-3 is closed source.
| varispeed wrote:
| If we feed AI all the knowledge about the physics of the world,
| then will it be ever capable of giving answers without actually
| performing scientific research inferring it just from the laws
| that define the world?
| supermatt wrote:
| Even on hard, if you understand the terminology, the fake ones
| are mostly gibberish.
| aidenn0 wrote:
| Right, I could pick out all the hard ones for fields I'm
| comfortable with, but struggled for some of the easy ones in
| fields I was less familiar with.
| oceliker wrote:
| If you don't understand the terminology, any paper is gibberish
| :) but I agree, I can detect fake ones fairly reliably in
| biology, but not in e.g. astronomy.
| karagenit wrote:
| Seems like the model likes to repeat words in the title,
| particularly when hyphens are involved (I guess it considers them
| as different words?) e.g. "new dinosaur-like dinosaur" and "male-
| pattern traits in male rats" are a couple I saw.
| minimaxir wrote:
| That's mostly a GPT-2/Transformers quirk. Some approaches apply
| a repetition penalty to work around it.
| dougb5 wrote:
| The generated abstracts may be gibberish but I wonder how often
| they contain little bits of brilliance, or make novel connections
| between ideas expressed in the training set. If we got a panel of
| domain experts to evaluate the snippets on this basis, thrir
| labels could be used to fine-tune the model in the direction of
| novel discovery. (This is almost certainly not a novel idea!)
| NorwegianDude wrote:
| Not hard at all. The fake ones doesn't make any sense from an
| English perspective. Looks like someone just picked the next word
| on a SwiftKey keyboard or something. "this word fits here...
| Right?"
| codeflo wrote:
| 5/5 on hard node, but it's tough sometimes, I don't actually know
| much about biology. But if you've played around with GPT before,
| you get better at spotting the subtle logical errors it tends to
| make. I wonder whether the ability to identify machine generated
| texts will become a useful skill at some point.
| minkzilla wrote:
| Or you just train a machine to do it and then generate a bunch
| and have this second machine sort out any it thinks are machine
| generated.
| toxik wrote:
| You basically just described a GAN. Neat!
| minimaxir wrote:
| GANs work by feeding back the mistakes and forcing the
| generator model to improve its cheating. In this case,
| filtering out titles that are ambiguous would act as an
| independent filter.
| truth_ wrote:
| The best fake-detecting model detecting fakes generated by
| the best generator model will always lag behind the latter
| model.
| minkzilla wrote:
| I think I see what you're saying, but why is this so?
| PeterisP wrote:
| In essence detecting which one is fake is a common way
| how you train the generator, tweaking the generating
| process to "fix" any detectable flaw; and you train it
| until (as far as your system is concerned) the generated
| texts are indistinguishable from the real ones. A better
| system might distinguish them, but that better system can
| be relatively trivially adapted to generate better texts
| which it won't be able to distinguish from real ones.
| f430 wrote:
| Progressively got tougher. I'm scared of the implications in like
| 20 years.
| drenvuk wrote:
| The hard version usually requires me to understand why a number,
| measurement or chemical or other substance doesn't make sense in
| the context of what each paragraph is describing. This means I
| can't just skim it in order to spot the fake, I need to figure
| out that what it's saying is wrong.
|
| That's close enough for this to be a success if the purpose was
| to persuade or fool laymen.
| lkbm wrote:
| This was a hard-mode fake I just got:
|
| > A new era of hyperaridididididemia revealed by single-cell
| RNA-seq
|
| So some are better than others. :-)
| neltnerb wrote:
| I also got one that was a fake that was talking about
| measuring two actual quantum properties simultaneously by
| using a third state to probe it indirectly.
|
| Which is absolutely a real thing except that the exact
| quantum properties in fact didn't commute while they claimed
| they did commute and said for some reason simultaneous
| measurement required a third state anyway.
|
| I don't know how I would have been able to distinguish that
| from completely reasonable methods for quantum error
| correction without knowing ahead of time which quantum states
| commute and which don't... pretty cool.
|
| If I were skimming or half asleep I definitely wouldn't have
| caught a lot of these on hard, abstracts are always so poorly
| written and usually trying too hard to be complicated
| sounding by using big words when small ones would do just
| fine!
| CSDude wrote:
| How does one train GPT-2 with their own content and produce nice
| results at arbitrary lengths? I found a few libraries but I could
| not use them well, I get lost very quickly. I just want to train
| our internal Confluence and have fun with it.
| th0ma5 wrote:
| Now here is an application where the GPT stuff can really shine,
| which is trying to convince people that aren't domain experts
| that something is speaking from authority, even if the reader
| doesn't intend to get anything meaningful from the material
| either way.
| hazeii wrote:
| On easy, 7 correct and 0 wrong was enough to tell me that yes, I
| can (I have been reading Nature for years though).
| zwaps wrote:
| Same here for hard mode.
|
| Still interesting!
| caslon wrote:
| On easy, ten correct and zero wrong was enough for me to
| realize the same (I have never read Nature).
|
| Trying hard right now, will report results after.
|
| Edit: Yeah, it's the same deal. Length becomes less of a
| giveaway but its errors become more obvious.
| anon_tor_12345 wrote:
| STEM people love to bring up the sokal affair. the same STEM
| people also don't realize that many journals and conferences in
| STEM have been tricked by things like this (more specifically
| precursors using HMMs and etc).
|
| https://en.wikipedia.org/wiki/List_of_scholarly_publishing_s...
|
| edit: don't understand why i'm getting downvoted. is my comment
| not relevant to a post about the plausibility of abstracts
| generated by ML models?
| Mordisquitos wrote:
| The reason for the downvotes is probably the generalisation
| regarding what _" STEM people love to bring up"_ and also _"
| don't realize"_. It feels like an unprovoked strawman attack
| against an ambiguously defined group of people.
| writeslowly wrote:
| I found it relatively easy to spot the fakes, but the titles on
| some of them were pretty good and made me wish they were real.
| Like reading science journals from a whimsical fantasy universe.
|
| Some of my favorites: "A new onset of primeval black magic in
| magic-ring crystals"
|
| "The genetic network for moderate religiosity in one thousand
| bespectacled twins"
|
| "Thermal vestige of the '70s and '00s disco ball trend"
| ronsor wrote:
| Even hard mode isn't _that_ hard because GPT-2 tends to ramble on
| while saying nothing substantive. If I can 't figure out what a
| paper is supposed to be talking about, it's fake.
|
| 4/4 on hard. Never read a Nature paper before.
| meowface wrote:
| (Easy/cliched joke, but)
|
| >If I can't figure out what a paper is supposed to be talking
| about, it's fake.
|
| Depends on the field...
| thereddaikon wrote:
| The engineering/materials science/physics ones were fairly
| easy to identify for me. Usually it would be one or two
| sentences that were grammatically cohesive but would make a
| statement that didn't make any sense if you had even a basic
| understanding of the topic. One that stood out to me was an
| astrophysics paper that said a planet was orbiting solar
| wind. I don't have to be a PhD to know that's BS.
|
| The medical and biotech ones are much harder.
| jacquesm wrote:
| Yes, this mirrors my experience. Fields that have my
| interest are pretty easy in isolation (just looking at one
| subject), but fields that are remote can be a challenge.
| hervature wrote:
| In this instance, the medical and biotech generated stuff
| are much harder to identify because the algorithm doesn't
| need to introduce grammar issues. For instance, here is a
| random paper abstract that I changed, can you spot the
| change? Hint, it is one of the Greek symbols or a number.
|
| Three highly pathogenic b-coronaviruses have crossed the
| animal-to-human species barrier in the past two decades:
| SARS-CoV, MERS-CoV and SARS-CoV-2. To evaluate the
| possibility of identifying antibodies with broad
| neutralizing activity, we isolated a monoclonal antibody,
| termed B4, that cross-reacts with eight b-coronavirus
| spike glycoproteins, including all five human-infecting
| b-coronaviruses.
| argvargc wrote:
| Did way better on Hard mode than Easy. I think people get bored
| of doing this before we can see real indicative results.
|
| Scores under 5 on what amounts to a coin flip doesn't strike me
| as so remarkable, especially when coupled with an incentivised
| reporting-bias as we see here. ("I got a high score! Proud to
| share!" Vs. "I got a low score, or an even score and look at
| all the people reporting high scores, think I might keep it to
| myself")
|
| Being as it is, at this juncture, I think the AI may still have
| a chance to be strong with this one.
|
| Also, were the AI to do well consistently, I'd think it might
| say more about the external unfamiliarity with, and the
| internal prevalence of, field-specific scientific jargon, than
| any AI's or human's innate intelligence.
| ta988 wrote:
| I'm sure GPT2 abstracts would fly through many conferences
| screening processes. I've seen talks and posters that were utter
| non-sense but everybody was too polite to say anything to the
| person or advisors.
|
| I've reviewed articles that were completely made up and the other
| reviewer didnt even detect that. Nor did the editor.
|
| I've contacted editors about utterly wrong papers, criticized the
| article on pubpeer, and the article is still published... Because
| it would harm their notoriety. Thats one of the madenning
| ascpects of academic publishing.
| matthewdgreen wrote:
| There is a long tail of weak journals in just about any field.
| When you think about it, this is inevitable in any society that
| has freedom of the press and where there exist incentives
| (evaluated by non-experts) for publishing. You have to evaluate
| journals the same way that you would evaluate products
| purchased in a flea market.
| riquito wrote:
| With discretion, php/bootstrap/jquery still do their job for the
| presentation layer
| generalizations wrote:
| I'm curious if the trained model is available. It would be very
| fun to play with.
| jacquesm wrote:
| The side-by-side display makes it pretty easy to distinguish the
| one from the other, simply compare them at a level where the one
| that makes the least sense is the one that is nonsense. Like that
| I score 10/11. But when looking at just the left side one
| suddenly the problem is _much_ harder, and I 'm happy to get
| better than even. Bits that don't help: not an English native
| writer. Seen too many real life papers with crappy writing that
| quite a few of these look plausible, especially when they are
| about fields that I know very little of.
|
| Presumably when you're a native English speaker _and_ have a
| broader interest the difficulty goes down a bit.
|
| I like this project very much and would like to see some overall
| scores, and it might not hurt to allow for a verified result link
| to detect bragging rather than actual results (not that anybody
| on HN would ever brag about their score ;) ).
|
| Overall: I'm not worried that generated papers will swamp the
| publications any day soon but for spam/click farms this must be a
| godsend and for sure it will cause trouble for search engines to
| classify real content from generated content.
| zitterbewegung wrote:
| I tried making fake tweets using GPT-2 two years ago. When I
| actually interviewed people to verify my model I got good
| results for people who didn't actively engage with twitter
| versus people who regularly engaged in twitter (note that this
| was an N ~ 10 people and it was limited to GPT-2 774M.
|
| I found that people would also refuse the test and would
| believe whatever the output of the model was due to my choice
| of subject.
|
| Others that did a similar exercise and tried to verify their
| results using reddit had a great deal of people who would be
| able to spot fakes quite easily.
|
| The biggest issue would be someone using a system to
| deliberately fool a targeted set of people which is easy given
| how ad networks are run.
| lainga wrote:
| Having read your comment first (ooh, horribile dictu on HN) I
| decided to try playing by only looking at the left paper and
| deciding if it was fake. Luckily the model seems to have picked
| up that "last names can be units" too strongly and the 2nd fake
| paper was discussing a frequency of "10 Jones".
| hutzlibu wrote:
| Nice advanced logic riddles.
| cblconfederate wrote:
| It seems some of the fake ones could easily have been real (e.g.
| the one about the 3d structure of bound Ach receptor) . I guess
| the brevity of the text helps to make it make sense and to make
| it indistinguishable
| carbocation wrote:
| Easy mode is cake.
|
| Hard mode is good enough that I'd like to see some sort of
| distance metric to the nearest real story, to be sure the model
| isn't accidentally copying truth.
| jandrese wrote:
| Sometimes the articles are really short which makes it even
| harder to figure out which is fake. GPT's big weakness is that
| it tends to forget what it was talking about and wanders off
| after a couple of paragraphs. With just one sentence to examine
| it can be very hard to spot.
| kurthr wrote:
| Yes, I got a really short astronomical one about the discovery
| of a metallic core planet circling a G-Type star, and I only
| knew it was fake, because I would have heard about it!
| quantum_mcts wrote:
| I always wanted similar thing but for some philosophy texts.
| Notably Hegel - I'd love to see a philosopher trying to figure
| out which pile of gibberish is generated and which is the work of
| a father of modern dialectics.
| beforeolives wrote:
| Cool demo.
|
| With these GPT models, I don't get the appeal of creating fake
| text that at best can pass as real to someone who doesn't
| understand the topic and context. What's the use case? Generating
| more believable spam for social media? Anything else? Because
| there's no real knowledge representation or information
| extraction going on here.
| minimaxir wrote:
| For _fun_.
| aabhay wrote:
| Fun? Show me the fun!
| finin wrote:
| We've done recent work on using a transformer to generate fake
| cyber threat intelligence (CTI) and found that a set of
| cybersecurity experts could not reliably distinguish the fake CTI
| examples from real ones.
|
| Priyanka Ranade, Aritran Piplai, Sudip Mittal, Anupam Joshi, and
| Tim Finin, Generating Fake Cyber Threat Intelligence Using
| Transformer-Based Models, Int. Joint Conf. on Neural Networks,
| IEEE, 2021. https://ebiq.org/p/969
| f6v wrote:
| The sad thing is that often there's an equal mental effort to
| read GPT articles and the real ones. It's as if people are trying
| to make their papers as incomprehensible as possible.
| FredPret wrote:
| Incomprehensible language = look how smart I am now give me
| grant money
| mattkrause wrote:
| Nah--If you're publishing in Nature, you're already well
| beyond that game.
|
| The incomprehensibility comes from the fact that abstracts
| (and particularly NPG abstracts) are trying to do many things
| at once--and all in 200 words. In theory, the abstract should
| describe why your work is of broad general interest (so
| Nature's editors will publish it), while explaining the
| specific scientific question and answer(!) to a specialist
| audience of often-picky, sometimes-hostile peer reviewers,
| and conforming to a fairly specific style that doesn't
| reference the rest of the paper.
|
| It's tough to do well, and even moreso for non-native English
| speakers.
| kangalioo wrote:
| At least in the samples I was presented, the more
| comprehensible articles were consistently the fake ones.
| jvanderbot wrote:
| Quite easy when you know one is fake. Flagging fake articles in a
| review queue, by abstract only, and when none may exist all the
| way up to all being fake ....
|
| Now that's a challenge.
|
| Also, if you train GPT on the whole corpus of Nature / Science /
| whatever articles up to, say, 2005, could you feed it leading
| text about discoveries after 2005 and see if it hypothesizes the
| justification for those discoveries in the same way that the
| authors did?
| jcims wrote:
| Excellent point. Serializing them would make it more difficult.
| NorwegianDude wrote:
| Considering the terrible quality of the writing in these
| examples, it's simple no matter how it's presented.
| pjc50 wrote:
| I find the whole thing ominous because there is no "there"
| there: there is no understanding in the GPT-2 system, but it's
| able to generate increasingly plausible text. This greatly
| increases the amount of plausible nonsense that can be used to
| drown out actual research. You could certainly replace a lot of
| pop-sci and start several political movements with GPT-2... all
| of which has no actual nutritional content.
| jvanderbot wrote:
| I find it intriguing for exactly that reason. I agree there's
| no fundamental "there", but I suggest that you may find that
| ominous because it implies there's no fundamental
| understanding anywhere. Only stories that survive scrutiny.
|
| GPT can write "about" something from a prompt. This is not
| much different than me interpreting data that I'm analyzing.
| I'm constantly generating stories and checking them, until
| one story survives it all. How do I generate stories!?
| Seriously. I'm sure I have a GPT module in my left frontal
| cortex. I use it all the time when I think about actions I
| take, and it's what I try to ignore when I meditate. Its
| ongoing narrative is what feeds back into how I feel about
| things, which affects how I interact with things and what
| things I interact with ... not necessarily as a goal-driven
| decision process, more as a feedback-driven randomized
| selection. Isn't this kind of the basis of Cognitive
| Behavioral Therapy, meditation, etc. See [1,2]. If you stick
| GPT and sentiment analysis into a room, will they produce a
| rumination feedback like a depressed person?
|
| Anyway, if you can tell a coherent story to justify a result
| (once presented with a result), one that is convincing enough
| for people to believe and internalize the result in their
| future studies, how is that different from understanding that
| result and teaching it to others? The act of teaching is
| itself story generation. Mental models are just story-driven
| hacks that allow people to generalize results in an insanely
| complex system.
|
| 1. Happiness Hypothesis Jonathan Haidt 2. Buddhism and modern
| psychology, coursera
| jrumbut wrote:
| You could probably ask an undergraduate or pop science fan
| to write a paper title and first sentence of an abstract
| that would turn heads and get good results.
|
| Faking an entire 10 page paper with figures and citations
| is much harder. I'm sure it'll happen next week, but until
| then I can still say that's where real understanding is
| demonstrated.
| fock wrote:
| this is already happening. maybe not in Nature, but all over
| the "small" papers of this world (and so also with any Nature
| derivatives). There's people milling the same article with
| nonsense "results" through 3 journals each in Elsevier,
| Springer, Wiley and RCS. And they stay up. Forever. And
| anyone looking into this topic will waste an hour because
| these 10x cited articles are just self-citations of the same
| crap.
| sdenton4 wrote:
| "there is no understanding in the GPT-2 system, but it's able
| to generate increasingly plausible text."
|
| So, basically it's achieved undergraduate level skills.
| brokencode wrote:
| Absolutely. Especially once you factor in the fact that
| sometimes things are simply written poorly or don't make sense,
| even if it was a real human being at the keyboard.
|
| A lot of the communication I have with folks is subtly flawed
| in logic or grammar, but that doesn't make me think I'm working
| with a bunch of androids.
|
| It's natural and often even necessary to try to figure out an
| author's intent when their writing doesn't fully make sense.
| dilippkumar wrote:
| Good point.
|
| This challenge would be more interesting if there were "Neither
| is fake" and "Both are fake" buttons (and obviously, the test
| randomly showed two fake and two real articles in the mix)
| blt wrote:
| I wish there was a version of this for computer science. We don't
| have a broad flagship journal like Nature, so maybe it would need
| to be trained on a collection of IEEE and ACM venues.
___________________________________________________________________
(page generated 2021-05-13 23:00 UTC)