[HN Gopher] GPTZero finds 100 new hallucinations in NeurIPS 2025...
___________________________________________________________________
GPTZero finds 100 new hallucinations in NeurIPS 2025 accepted
papers
Author : segmenta
Score : 624 points
Date : 2026-01-22 15:20 UTC (7 hours ago)
(HTM) web link (gptzero.me)
(TXT) w3m dump (gptzero.me)
| cogman10 wrote:
| Yuck, this is going to really harm scientific research.
|
| There is already a problem with papers falsifying
| data/samples/etc, LLMs being able to put out plausible papers is
| just going to make it worse.
|
| On the bright side, maybe this will get the scientific community
| and science journalists to finally take reproducibility more
| seriously. I'd love to see future reporting that instead of
| saying "Research finds amazing chemical x which does y" you see
| "Researcher reproduces amazing results for chemical x which does
| y. First discovered by z".
| godzillabrennus wrote:
| Have they solved the issue where papers that cite research
| already invalidated are still being cited?
| cogman10 wrote:
| AFAIK, no, but I could see there being cause to push
| citations to also cite the validations. It'd be good if
| standard practice turned into something like
|
| Paper A, by bob, bill, brad. Validated by Paper B by carol,
| clare, charlotte.
|
| or
|
| Paper A, by bob, bill, brad. Unvalidated.
| gcr wrote:
| Academics typically use citation count and popularity as a
| rough proxy for validation. It's certainly not perfect, but
| it is something that people think about. Semantic Scholar
| in particular is doing great work in this area, making it
| easy to see who cites who: https://www.semanticscholar.org/
|
| Google Scholar's PDF reader extension turns every
| hyperlinked citation into a popout card that shows citation
| counts inline in the PDF:
| https://chromewebstore.google.com/detail/google-scholar-
| pdf-...
| rtkwe wrote:
| That is a factor most people miss when thinking about the
| replication crisis. For the harder physical sciences a
| wrong paper will fairly quickly be found because as
| people go to expand on the ideas/use that data and get
| results that don't match the model informed by paper X
| they're going to eventually figure out that X is wrong.
| There might be issues with getting incentives to write
| and publish that negative result but each paper where the
| results of a previous paper are actually used in the new
| paper is a form of replication.
| reliabilityguy wrote:
| Nope.
|
| I am still reviewing papers that propose solutions based on a
| technique X, conveniently ignoring research from two years
| ago that shows that X cannot be used on its own. Both the
| paper I reviewed and the research showing X cannot be used
| are in the same venue!
| b00ty4breakfast wrote:
| does it seem to be legitimate ignorance or maybe folks
| pushing ahead regardless of x being disproved?
| freedomben wrote:
| IMHO, It's mostly ignorance coming a push/drive to
| "publish or perish." When the stakes are so high and
| output is so valued, and when reproducability isn't
| required, it disincentivizes thorough work. The system is
| set up in a way that is making it fail.
|
| There is also the reality that "one paper" or "one study"
| can be found contradicted almost anything, so if you just
| went with "some other paper/study debunks my premise"
| then you'd end up producing nothing. Plus many inside
| know that there's a lot of slop out there that gets
| published, so they can (sometimes reasonably IMHO)
| dismiss that "one paper" even when they do know about it.
|
| It's (mostly) not fraud or malicious intent or ignorance,
| it's (mostly) humans existing in the system in which they
| must live.
| reliabilityguy wrote:
| Poor scholarship.
|
| However, given the feedback by other reviewers, I was the
| only one who knew that X doesn't work. I am not sure how
| these people mark themselves as "experts" in the field if
| they are not following the literature themselves.
| f311a wrote:
| For ML/AI/Comp sci articles, providing reproducible code is a
| great option. Basically, PoC or GTFO.
| StableAlkyne wrote:
| The most annoying ones are those which discuss loosely the
| methodology but then fail to publish the weights or any real
| algorithms.
|
| It's like buying a piece of furniture from IKEA, except you
| just get an Allen key, a hint at what parts to buy, and
| blurry instructions.
| j45 wrote:
| It will better expose the behaviour of false scientists.
| StableAlkyne wrote:
| > I'd love to see future reporting that instead of saying
| "Research finds amazing chemical x which does y" you see
| "Researcher reproduces amazing results for chemical x which
| does y. First discovered by z".
|
| Most people (that I talk to, at least) in science agree that
| there's a reproducibility crisis. The challenge is there really
| isn't a good way to incentivize that work.
|
| Fundamentally (unless you're independent wealthy and funding
| your own work), you have to measure productivity _somehow,_
| whether you 're at a university, government lab, or the private
| sector. That turns out to be very hard to do.
|
| If you measure raw number of papers (more common in developing
| countries and low-tier universities), you incentivize a flood
| of junk. Some of it is good, but there is such a tidal wave of
| shit that most people write off your work as a heuristic based
| on the other people in your cohort.
|
| So, instead it's more common to try to incorporate how "good" a
| paper is, to reward people with a high quantity of "good"
| papers. That's quantifying something subjective though, so you
| might try to use something like citation count as a proxy: if a
| work is impactful, usually it gets cited a lot. Eventually you
| may arrive at something like the H-index, which is defined as
| "The highest number H you can pick, where H is the number of
| papers you have written with H citations." Now, the trouble
| with this method is people won't want to "waste" their time on
| incremental work.
|
| And that's the struggle here; even if we funded and rewarded
| people for reproducing results, they will always be bumping up
| the citation count of the original discoverer. But it's worse
| than that, because literally nobody is going to cite your work.
| In 10 years, they just see the original paper, a few citing
| works reproducing it, and to save time they'll just cite the
| original paper only.
|
| There's clearly a problem with how we incentivize scientific
| work. And clearly we want to be in a world where people test
| reproducibility. However, it's very very hard to get there when
| one's prestige and livelihood is directly tied to discovery
| rather than reproducibility.
| warkdarrior wrote:
| > If you measure raw number of papers (more common in
| developing countries and low-tier universities), you
| incentivize a flood of junk.
|
| This is exactly what rewarding replication papers (that
| reproduce and confirm an existing paper) will lead to.
| pixl97 wrote:
| And yet if we can't reproduce an existing paper, it's very
| possible that existing paper is junk itself.
|
| Catch-22 is a fun game to get caught in.
| maerF0x0 wrote:
| > The challenge is there really isn't a good way to
| incentivize that work.
|
| What if we got Undergrads (with hope of graduate studies) to
| do it? Could be a great way to train them on the skills
| required for research without the pressure of it also being
| novel?
| StableAlkyne wrote:
| Those undergrads still need to be advised and they use lab
| resources.
|
| If you're a tenure-track academic, your livelihood is much
| safer from having them try new ideas (that you will be the
| corresponding author on, increasing your prestige and
| ability to procure funding) instead of incrementing.
|
| And if you already have tenure, maybe you have the
| undergrad do just that. But the tenure process heavily
| filters for ambitious researchers, so it's unlikely this
| would be a priority.
|
| If instead you did it as coursework, you could get them to
| maybe reproduce the work, but if you only have the students
| for a semester, that's not enough time to write up the
| paper and make it through peer review (which can take
| months between iterations)
| suddenlybananas wrote:
| Unfortunately, that might just lead to a bunch of type II
| errors instead, if an effect requires very precise
| experimental conditions that undergrads lack the expertise
| for.
| retsibsi wrote:
| Could it be useful as a first line of defence? A failed
| initial reproduction would not be seen as disqualifying,
| but it would bring the paper to the attention of more
| senior people who could try to reproduce it themselves.
| (Maybe they still wouldn't bother, but hopefully they'd
| at least be more likely to.)
| rtkwe wrote:
| Most interesting results are not so simple to recreate that
| would could reliably expect undergrads to do perform the
| replication even if we ignore the cost of the equipment and
| consumables that replication would need and the
| time/supervision required to walk them through the process.
| jimbokun wrote:
| > The challenge is there really isn't a good way to
| incentivize that work.
|
| Ban publication of any research that hasn't been reproduced.
| wpollock wrote:
| > Ban publication of any research that hasn't been
| reproduced.
|
| Unless it is published, nobody will know about it and thus
| nobody will try to reproduce it.
| sroussey wrote:
| Just have a new journal of only papers that have been
| reproduced, and include the reproduction papers.
| gcr wrote:
| lol, how would the first paper carrying some new discovery
| get published?
| dekhn wrote:
| If we did that, CERN could not publish, because nobody else
| has the capabilities they do. Do we really want to punish
| CERN (which has a good track record of scientific
| integrity) because their work can't be reproduced? I think
| the model in many of these cases is that the lab publishing
| has to allow some number of postdocs or competitor labs to
| come to their lab and work on reproducing it in-house with
| the same reagents (biological experiments are remarkably
| fragile).
| poulpy123 wrote:
| > I'd love to see future reporting that instead of saying
| "Research finds amazing chemical x which does y" you see
| "Researcher reproduces amazing results for chemical x which
| does y. First discovered by z".
|
| But nobody want to pay for it
| geokon wrote:
| usually you reproduce previous research as a byproduct of
| doing something novel "on top" of the previous result. I dont
| really see the problem with the current setup.
|
| sometimes you can just do something new and assume the
| previous result, but thats more the exception. youre almost
| always going to at least in part reproducr the previous one.
| and if issues come up, its often evident.
|
| thats why citations work as a good proxy. X number of people
| have done work based around this finding and nobody has seen
| a clear problem
|
| theres a problem of people fabricating and fudging data and
| not making their raw data available ("on request" or with not
| enough meta data to be useful) which wastes everyones time
| and almost never leads to negative consequences for the
| authors
| gcr wrote:
| It's often quite common to see a citation say "BTW, we
| weren't able to reproduce X's numbers, but we got fairly
| close number Y, so Table 1 includes that one next to an
| asterisk."
|
| The difficult part is surfacing that information to readers
| of the original paper. The semantic scholar people are
| beginning to do some work in this area.
| geokon wrote:
| yeah thats a good point. the citation might actually be
| pointing out a problem and not be a point in favor. its a
| slog to figure out... but seems like the exact type of
| problem an LLM could handle
|
| give it a published paper and it runs through papers that
| have cited it and give you an evaluation
| gcr wrote:
| I'd personally like to see top conferences grow a
| "reproducibility" track. Each submission would be a short
| tech report that chooses some other paper to re-implement.
| Cap 'em at three pages, have a lightweight review process.
| Maybe there could be artifacts (git repositories, etc) that
| accompany each submission.
|
| This would especially help newer grad students learn how to
| begin to do this sort of research.
|
| Maybe doing enough reproductions could unlock incentives.
| Like if you do 5 reproductions than the AC would assign your
| next paper double the reviewers. Or, more invasively, maybe
| you can't submit to the conference until you complete some
| reproduction.
| azan_ wrote:
| The problem is that reproducing something is really, really
| hard! Even if something doesn't reproduce in one
| experiment, it might be due to slight changes in some
| variables we don't even think about. There are some ways to
| circumvent it (e.g. team that's being reproduced
| cooperating with reproducing team and agreeing on what
| variables are important for the experiemnt and which are
| not), but it's really hard. The solutions you propose will
| unfortunately incentivize bad reproductions and we might
| reject theories that are actually true because of that. I
| think that one of the best way to fight the crisis is to
| actually improve quality of science - articles where
| authors reject to share their data should be automatically
| rejected. We should also move towards requiring
| preregistration with strict protocols for almost all
| studies.
| AnIrishDuck wrote:
| Yeah, this feels like another reincarnation of the
| ancient "who watches the watchmen?" problem [1]. Time and
| time again we see that the incentives _really really_
| matter when facing this problem; subtle changes can
| produce entirely new problems.
|
| 1. https://en.wikipedia.org/wiki/Quis_custodiet_ipsos_cus
| todes%...
| gcr wrote:
| That's fine! The tech report should talk about what the
| researchers tried and what didn't work. I think
| submissions to the reproducibility track shouldn't
| necessarily have to be positive to be accepted, and
| conversely, I don't think the presence of a negative
| reproduction should necessarily impact an author's career
| negatively.
| gowld wrote:
| Every time some easy "Reproducibility is hard / not worth
| the effort" I hear "The original research wasn't
| meaningful or valuable".
| azan_ wrote:
| And that's true! It doesn't make sense to spend a lot of
| resources on reproducing things when there is low hanging
| fruit of just requiring better research in the first
| place.
| dataflow wrote:
| Is it time for some sort of alternate degree to a PhD
| beyond a Master's? Showing, essentially, "this person can
| learn, implement, validate, and analyze the state of the
| art in this field"?
| gogopromptless wrote:
| Thats what we call a Staff level engineer. Proven ability
| to learn, implement and validate is basically the "it
| factor" businesses are looking for.
|
| If you are thinking about this from an academic angle
| then sure its sounds weird to say "Two Staff jobs in a
| row from the University of LinkedIn" as a degree. But I
| submit this as basically the certificate you desire.
| dataflow wrote:
| No, this is not at all being a staff engineer. One is
| about delivering high-impact projects toward a business's
| needs, with all the soft/political things that involves,
| and the other is about implementing and validating
| cutting-edge research, with all the deep academic and
| technical knowledge and work that that involves. They're
| incredibly different skillsets, and many people doing one
| would easily fail in the other.
| gcr wrote:
| That sounds precisely like the function of a Ph.D. to me.
| dataflow wrote:
| A PhD is for making new contributions to a field, not
| validating existing ones.
| MetaWhirledPeas wrote:
| > Eventually you may arrive at something like the H-index,
| which is defined as "The highest number H you can pick, where
| H is the number of papers you have written with H citations."
|
| It's the Google search algorithm all over again. And it's the
| certificate trust hierarchy all over again. We keep working
| on the same problems.
|
| Like the two cases I mentioned, this is a matter of making
| adjustments until you have the desired result. Never perfect,
| always improving (well, we hope). This means we need
| liquidity with the rules and heuristics. How do we best get
| that?
| sroussey wrote:
| Incentives.
|
| First X people that reproduce Y get Z percent of patent
| revenue.
|
| Or something similar.
| wizzwizz4 wrote:
| I'm delighted to inform you that I have reproduced every
| patent-worthy finding of every major research group
| active in my field in the past 10 years. You can check my
| data, which is exactly as theory predicts (subject to
| some noise consistent with experimental error). I accept
| payment in cash.
| jltsiren wrote:
| Patent revenue is mostly irrelevant, as it's too
| unpredictable and typically decades in the future.
| Academics rarely do research that can be expected to
| produce economic value in the next 10-20 years, because
| the industry can easily outspend the academia in such
| topics.
| rtkwe wrote:
| Most papers generate zero patent revenue or even lead to
| patents at all. For major drugs maybe that works but we
| already have clinical trials before the drug goes to
| market that validate the efficacy of the drugs.
| graemep wrote:
| > you have to measure productivity somehow,
|
| No, you do not have to. You give people with the skills and
| interest in doing research the money. You need to ensure its
| spent correctly, that is all. People will be motivated by
| wanting to build a reputation and the intrinsic reward of the
| work
| soiltype wrote:
| That feels arbitrary as a measure of quality. Why isn't new
| research simply devalued and replication valued higher?
|
| "Dr Alice failed to reproduce 20 would-be headline-grabbing
| papers, preventing them from sucking all the air out of the
| room in cancer research" is something laudable, but we're not
| lauding it.
| agumonkey wrote:
| I think, at least I hope, that a part of the LLM value will be
| to create their retirement for specific needs. Instead of
| asking it to solve any problem, restrict the space to a tool
| that can help you then reach your goal faster without the
| statistical nature of LLMs.
| mike_hearn wrote:
| Reproducibility is overrated and if you could wave a wand to
| make all papers reproducible tomorrow, it wouldn't fix the
| problem. It might even make it worse.
|
| https://blog.plan99.net/replication-studies-cant-fix-science...
| biophysboy wrote:
| ? More samples reduces the variance of a statistic. Obviously
| it cannot identify systematic bias in a model, or establish
| causality, or make a "bad" question "good". Its not overrated
| though -- it would strengthen or weaken the case for many
| papers.
| mike_hearn wrote:
| If you have a strong grip on exactly what it means, sure,
| but look at any HN thread on the topic of fraud in science.
| People think replication = validity because it's been
| described as the replication crisis for the last 15 years.
| And that's the best case!
|
| Funding replication studies in the current environment
| would just lead to lots of invalid papers being promoted as
| "fully replicated" and people would be fooled even harder
| than they already are. There's got to be a fix for the
| underlying quality issues before replication becomes the
| next best thing to do.
| doctorpangloss wrote:
| while i agree that "reproducibility is overrated", i went
| ahead and read your medium post. my feedback to you is,
| my summary of that writing: "mike_hearn's take on policy-
| adjacent writing conducted by public health officials and
| published in journals that interacted with mike_hearn's
| valid and common but nonetheless subjective political
| dispute about COVID-19."
|
| i don't know how any of that writing generalizes to other
| parts of academic research. i mean, i know that you say
| it does, but i don't think it does. what exactly do you
| think most academic research institutions and the federal
| government spend money on? for example, wet lab research.
| you don't know anything about wet lab research. i think
| if you took a look at a typical e.g. basic science in
| immunology paper, built on top of mouse models, you would
| literally lose track of any of its meaning after the
| first paragraph, you would feed it into chatgpt, and you
| would struggle to understand the topic well enough to
| read another immunology paper, you would have an immense
| challenge talking about it with a researcher in the
| field. it would take weeks of reading. you have no
| medicine background, so you wouldn't understand the long
| horizon context of any of it. you wouldn't be able to
| "chatbot" your way into it, it would be a real education.
| so after all of that, would you still be able to write
| the conclusion you wrote in the medium post? i don't
| think so, because you would see that by many measures,
| you _cannot_ generalize a froo-froo policy between
| "subjective political dispute about COVID-19" writing and
| wet lab research. you'd gain the wisdom to see that
| they're different things, and you lack the background,
| and you'd be much more narrow in what you'd say.
|
| it doesn't even have to be in the particulars, it's just
| about wisdom. that is my feedback. you are at once saying
| that there is greater wisdom to be had in the
| organization and conduct of research, and then, you go
| and make the highly low wisdom move to generalize about
| _all_ academic research. which you are obviously doing
| not because it makes sense to, you 're a smart guy. but
| because you have some unknown beef with "academics" that
| stems from anger about valid, common but nonetheless
| subjective political disputes about COVID-19.
| mike_hearn wrote:
| Thanks for reading it, or scan reading it maybe. Of the
| 18 papers discussed in the essay here's what they're
| about in order:
|
| - Alzheimers
|
| - Cancer
|
| - Alzheimers
|
| - Skin lesions (first paper discussed in the linked blog
| post)
|
| - Epidemiology (COVID)
|
| - Epidemiology (COVID, foot and mouth disease, Zika)
|
| - Misinformation/bot studies
|
| - More misinformation/bot studies
|
| - Archaeology/history
|
| - PCR testing (in general, discussion opens with testing
| of whooping cough)
|
| - Psychology, twice (assuming you count "men would like
| to be more muscular" as a psych claim)
|
| - Misinformation studies
|
| - COVID (the highlighted errors in the paper are
| objective, not subjective)
|
| - COVID (the highlighted errors are software bugs, i.e.
| objective)
|
| - COVID (a fake replication report that didn't
| successfully replicate anything)
|
| - Public health (from 2010)
|
| - Social science
|
| Your summary of this as being about a "valid and common
| but subjective political dispute" I don't agree is
| accurate. There's no politics involved in any of these
| discussions or problems, just bad science.
|
| Immunology has the same issues as most other medical
| fields. Sure, there's _also_ fraud that requires
| genuinely deep expertise to find, but there 's plenty
| that doesn't. Here's a random immunology paper from a few
| days ago identified as having image duplications,
| Photoshopping of western blots, numerous irrelevant
| citations and weird sentence breaks all suggestive that
| the paper might have been entirely faked or at least
| partly generated by AI: https://pubpeer.com/publications/
| FE6C57F66429DE2A9B88FD245DD...
|
| The authors reply, claiming the problems are just rank
| incompetence, and each time someone finds yet another
| problem with the paper leading to yet another apology and
| proclamation of incompetence. It's just another day on
| PubPeer, nothing special about this paper. I plucked it
| off the front page. Zero wet lab experience is needed to
| understand why the exact same image being presented as
| two different things in two different papers is a
| problem.
|
| And as for other fields, they're often extremely shallow.
| I actually am an expert in bot detection but that doesn't
| help at all in detecting validity errors in social
| science papers, because they do things like define a bot
| as anyone who tweets five times after midnight from a
| smartphone. A 10 year old could notice that this isn't
| true.
| doctorpangloss wrote:
| it only takes one drop of talking about COVID to make it
| about politics haha
| biophysboy wrote:
| > look at any HN thread on the topic of fraud in science.
|
| HN is very tedious/lazy when it comes to science
| criticism -- very much agree with you on this.
|
| My only point is replication is necessary to establish
| validity, _even if_ it is not sufficient. Whether it
| gives a scientist a false sense of security doesn 't
| change the math of sampling.
|
| I also agree with you on quality issues. I think
| alternative investment strategies (other than project
| grants) would be a useful step for reducing perverse
| incentives, for example. But there's a lot of things
| science could do.
| vld_chk wrote:
| In my mental model, the fundamental problem of reproducibility
| is that scientists have very hard time to find a penny to fund
| such research. No one wants to grant "hey I need $1m and 2
| years to validate the paper from last year which looks
| suspicious".
|
| Until we can change how we fund science on the fundamental
| level; how we assign grants -- it will be indeed very hard
| problem to deal with.
| parpfish wrote:
| In theory, asking grad students and early career folks to run
| replications would be a great training tool.
|
| But the problem isn't just funding, it's time. Successfully
| running a replication doesn't get you a publication to help
| your career.
| iugtmkbdfil834 wrote:
| Yeah, but doesn't publishing an easily falsifiable paper
| end one?
| wizzwizz4 wrote:
| Not in most fields, unless misconduct is evident. (And
| what constitutes "misconduct" is cultural: if you have
| enough influence in a community, you can exert that
| influence on exactly where that definitional border
| lies.) Being wrong is not, and _should not be_ , a
| career-ending move.
| iugtmkbdfil834 wrote:
| If we are aiming for quality, then being wrong absolutely
| should be. I would argue that is how it works in real
| life anyway. What we quibble over is what is the
| appropriate cutoff.
| rtkwe wrote:
| There's a big gulf between being wrong because you or a
| collaborator missed an uncontrolled confounding factor
| and falsifying or altering results. Science accepts that
| people sometimes make mistakes in their work because a)
| they can also be expected to miss something eventually
| and b) a lot of work is done by people in training in
| labs you're not directly in control of (collaborators).
| They already aim for quality and if you're consistently
| shown to be sloppy or incorrect when people try to use
| your work in their own.
|
| The final bit is a thing I think most people miss when
| they think about replication. A lot of papers don't get
| replicated directly but their measurements do when other
| researchers try to use that data to perform their own
| experiments, at least in the more physical sciences this
| gets tougher the more human centric the research is. You
| can't fake or be wrong for long when you're writing
| papers about the properties of compounds and molecules.
| Someone is going to come try to base some new idea off
| your data and find out you're wrong when their experiment
| doesn't work. (or spend months trying to figure out
| what's wrong and finally double check the original data).
| wizzwizz4 wrote:
| In fields like psychology, though, you can be wrong for
| _decades_. If your result is foundational enough, and
| other people have "replicated" it, then most researchers
| will toss out contradictory evidence as "guess those
| people were an unrepresentative sample". This can be
| extremely harmful when, for instance, the prevailing view
| is "this demographic are just perverts" or "most humans
| are selfish thieves at heart, held back by perceived
| social consensus" - both examples where researcher
| misconduct elevated baseless speculation to the position
| of "prevailing understanding", which led to bad policy,
| which had devastating impacts on people's lives.
|
| (People are better about this _in psychology_ , now:
| schoolchildren are taught about some of the more
| egregious cases, even before university, and individual
| researchers are much more willing to take a sceptical
| view of certain suspect classes of "prevailing
| understanding". The fact that even I, a non-psychologist,
| know about this, is good news. But what of the fields
| whose practitioners _don 't_ know they have this
| problem?)
| rtkwe wrote:
| Yeah like I said the soft validation by subsequent papers
| is more true in more baseline physical sciences because
| it involves fewer uncontrollable variables. That's why I
| mentioned 'hard' sciences in my post, messy humans are
| messy and make science waaay harder.
| parpfish wrote:
| But the thing is... nobody is doing the replication to
| falsify it. And if the did, it wouldn't be published
| because it's a null result
| bnchrch wrote:
| One, it doesnt damage your reputation as much as one
| would think.
|
| But two, and more importantly, no one is checking.
|
| Tree falls in the forest, no one hears, yadi-yada.
| iugtmkbdfil834 wrote:
| << no one is checking.
|
| I think this is the big part of it. There is no incentive
| to do it even when the study can be reproduced.
| godelski wrote:
| Here's a work from last year which was plagiarized. The
| rare thing about this work is it was submitted to ICLR,
| which opened reviews for both rejected and accepted
| works.
|
| You'll notice you can click on author names and you'll
| get links to their various scholar pages but notably
| DBLP, which makes it easy to see how frequently authors
| publish with other specific authors.
|
| Some of those authors have very high citation counts...
| in the thousands, with 3 having over 5k each (one with
| over 18k).
|
| https://openreview.net/forum?id=cIKQp84vqN
| Telaneo wrote:
| Not really, since nobody (for values of) ends up actually
| falsifying it, and if they do, it's years down the line.
| m-schuetz wrote:
| The vast majority of papers is so insignifcant, nobody
| bothers to try and use and thereby replicate it.
| goalieca wrote:
| Grad students don't get to publish a thesis on
| reproduction. Everyone from the undergraduate research
| assistant to the tenured professor with research chairs are
| hyper focused on "publishing" as much "positive result" on
| "novel" work as possible
| Kinrany wrote:
| Publishing a replication could be a prerequisite to
| getting the degree
|
| The question is, how can universities coordinate to add
| this requirement and gain status from it
| ihaveajob wrote:
| I think Arxiv and similar could contribute positively by
| listing replications/falsifications, with credit to the
| validating authors. That would be enough of an incentive
| for aspiring researchers to start making a dent.
| bonoboTP wrote:
| Prerequisite required by who, and why is that entity
| motivated to design such a requirement? Universities also
| want more novel breakthrough papers to boast about and to
| outshine other universities in the rankings. And if one
| is honest, other researchers also get more excited about
| new ideas than a failed replication that may for a
| thousand different reasons and the original authors will
| argue you did something wrong, or evaluated in an unfair
| way, and generally publicly accusing other researchers of
| doing bad work won't help your career much. It's a small
| world, you'd be making enemies with people who will sit
| on your funding evaluation committees, hiring committees
| and it just generally leads to drama. Also papers are
| superseded so fast that people don't even care that a no
| longer state of the art paper may have been wrong. There
| are 5 newer ones that perform better and nobody uses the
| old one. I'm just stating how things actually are, I
| don't say that this is good, but when you say something
| "should" happen, think about who exactly is motivated to
| drive such a change.
| derektank wrote:
| > Prerequisite required by who, and why is that entity
| motivated to design such a requirement?
|
| Grant awarding institutions like the NIH and NSF
| presumably? The NSF has as one of its functions, "to
| develop and encourage the pursuit of a national policy
| for the promotion of basic research and education in the
| sciences". Encouraging the replication of research as
| part of graduate degree curricula seems to fall within
| bounds. And the government's interest in science isn't
| novelty per se, it's the creation and dissemination of
| factually correct information that can be useful to its
| constituents.
| bonoboTP wrote:
| The commenter I was replying to wanted it to be a
| prerequisite for a degree, not for a grant. Grant
| awarding institutions also have to justify their spending
| to other parts of government and/or parliament
| (specifically, politicians). Both politicians and the
| public want to see breakthrough results that have the
| potential to cure cancer and whatnot. They want to boast
| that their funding contributed to winning some big-name
| prize and so on. You have to think with the mind of
| specific people in specific positions and what makes them
| look good, what gets them praise, promotions and friends.
|
| > And the government's interest in science isn't novelty
| per se, it's the creation and dissemination of factually
| correct information that can be useful to its
| constituents.
|
| This sounds very naive.
| soiltype wrote:
| But that seems almost trivially solved. In software it's
| common to value independent verification - e.g. code
| review. Someone who is only focused on writing new code
| instead of careful testing, refactoring, or peer review
| is widely viewed as a shitty developer by their peers. Of
| course there's management to consider and that's where
| incentives are skewed, but we're talking about a
| different structure. Why wouldn't the following work?
|
| A single university or even department could make this
| change - reproduction is the important work, reproduction
| is what earns a PhD. Or require some split, 20-50% novel
| work maybe is also expected. Now the incentives are
| changed. Potentially, this university develops a
| reputation for reliable research. Others may follow suit.
|
| Presumably, there's a step in this process where money
| incentivizes the opposite of my suggestion, and I'm not
| familiar with the process to know which.
|
| Is it the university itself which will be starved of
| resources if it's not pumping out novel (yet
| unreproducible) research?
| worik wrote:
| > In software it's common to value independent
| verification - e.g. code review. Someone who is only
| focused on writing new code instead of careful testing,
| refactoring, or peer review is widely viewed as a shitty
| developer by their peers.
|
| That is good practice
|
| It is rare, not common. Managers and funders pay for
| features
|
| Unreliable insecure software sells very well, so making
| reliable secure software is a "waste of money", generally
| DSMan195276 wrote:
| > Presumably, there's a step in this process where money
| incentivizes the opposite of my suggestion, and I'm not
| familiar with the process to know which.
|
| > Is it the university itself which will be starved of
| resources if it's not pumping out novel (yet
| unreproducible) research?
|
| Researchers apply for grants to fund their research, the
| university is generally not paying for it and instead
| they receive a cut of the grant money if it is awarded
| (IE. The grant covers the costs to the university for
| providing the facilities to do the research). If a
| researcher could get funding to reproduce a result then
| they could absolutely do it, but that's not what funds
| are usually being handed out for.
| bonoboTP wrote:
| Universities are not really motivated to slow down the
| research careers of their employees, on the contrary.
| They are very much interested in their employees making
| novel, highly cited publications and bringing in grants
| that those publications can lead to.
| coryrc wrote:
| Enough people will falsify the replication and pocket the
| money, taking you back to where you were in the first place
| and poorer for it. The loss of trust is an existential
| problem for the USA.
| eks-reigh wrote:
| You may well know this, but I get the sense that it isn't
| necessarily common knowledge, so I want to spell it out
| anyway:
|
| In a lot of cases, the salary for a grad student or tech is
| small potatoes next to the cost of the consumables they use
| in their work.
|
| For example,I work for a lab that does a lot of sequencing,
| and if we're busy one tech can use 10k worth of reagents in
| a week.
| bonoboTP wrote:
| We are on the comment section about an AI conference and
| up until the last few years material/hardware costs for
| computer science research was very cheap compare to other
| sciences like medicine, biology etc. where they use
| bespoke instruments and materials. In CS, up until very
| recently, all you needed was a good consumer PC for each
| grad student that lasted for many years. Nowadays GPU
| clusters are more needed but funding is generally not
| keeping up with that, so even good university labs are
| way underresourced on this front.
| rtkwe wrote:
| That.. still requires funding. Even if your lab happens to
| have all the equipment required to replicate you're paying
| the grad student for their time spent on replicating this
| paper and you'll need to buy some supplies; chemicals,
| animal subjects, pay for shared equipment time, etc.
| bandrami wrote:
| Grad students have this weird habit of eating food and
| renting places to live, though, so that's also money
| poszlem wrote:
| I often think we should movefrom peer review as
| "certification" to peer review as "triage", with replication
| determining how much trust and downstream weight a result
| earns over time.
| pas wrote:
| grants should come with money and requirement for
| independent reproduction
|
| academia is too fragmented and extremely inefficient
| jghn wrote:
| Partially. There's also the issue that some sciences, like
| biology, are a lot messier & less predicatble than people
| like to believe.
| godelski wrote:
| Funding is definitely a problem, but frankly reproduction is
| common. If you build off someone else's work (as is the norm)
| you need to reproduce first.
|
| But without repetition being impactful to your career and the
| pressure to quickly and constantly push new work, a failure
| to reproduce is generally considered a reason to move on and
| tackle a different domain. It takes longer to trace the
| failure and the bar is higher to counter an existing work.
| It's much more likely you've made a subtle mistake. It's much
| more likely the other work had a subtle success. It's much
| more likely the other work simply wasn't written such that a
| work could be sufficiently reproduced.
|
| I speak from experience too. I still remember in grad school
| I was failing to reproduce a work that was the main
| competitor to the work I had done (I needed to create
| comparisons). I emailed the author and got no response.
| Luckily my advisor knew the author's advisor and we got a
| meeting set up and I got the code. It didn't do what was
| claimed in the paper and the code structure wasn't what was
| described either. The result? My work didn't get published
| and we moved on. The other work was from a top 10 school and
| the choice was to burn a bridge and put a black mark on my
| reputation (from someone with far more merit and prestige) or
| move on.
|
| That type of thing won't change in a reproduction system but
| needs an open system and open reproduction system as well.
| Mistakes are common and we shouldn't punish them. The only
| way to solve these issues is openness
| bandrami wrote:
| > If you build off someone else's work (as is the norm) you
| need to reproduce first.
|
| Not if the result you're building off of is a model, you
| can just assume it
| pas wrote:
| yes, this should be built-in to grants and publishing
|
| of course the problem is that academia likes to assert its
| autonomy (and grant orgs are staffed by academia largely)
| benob wrote:
| Maybe it will also change the whole publication as evaluation
| of science.
| lxgr wrote:
| > LLMs being able to put out plausible papers is just going to
| make it worse
|
| If correct form (LaTeX two-column formatting, quoting the right
| papers and authors of the year etc.) has been allowing
| otherwise reject-worthy papers to slip through peer review,
| academia arguably has bigger problems than LLMs.
| LPisGood wrote:
| Correct form and relevant citations have been, for
| generations up to a couple of years ago, mighty strong
| signals that a work is good and done by a serious and
| reliable author. This is no longer the case and we are worse
| off for it.
| CamperBob2 wrote:
| I'd need to see the same scrutiny applied to pre-AI papers. If
| a field has a poor replication rate, meaning there's a good
| chance that a given published paper is just so much junk
| science, is that better or worse than letting AI hallucinate
| the data in the first place?
| Sparkyte wrote:
| If there is one thing which scientific reports must require is
| not using AI to produce the documentation. They can be of the
| data but not of the source or anything else. AI is a tool, not
| a replacement for actual work.
| lallysingh wrote:
| On the bright side, an LLM can really help set up a
| reproduction environment.
|
| Perhaps repro should become the basis of peer review?
| mort96 wrote:
| No, it can't. No LLM can purchase the equipment and chemicals
| and machinery you need to reproduce experiments, nor should
| you want it.
| colechristensen wrote:
| Reading the article, this is about CITATIONS which are
| trivially verifiable.
|
| This is just article publishers not doing the most basic
| verification failing to notice that the citations in the
| article don't exist.
|
| What this should trigger is a black mark for all of the authors
| and their institutions, both of which should receive
| significant reputational repercussions for publishing fake
| information. If they fake the easiest to verify information
| (does the cited work exist) what else are they faking?
| godelski wrote:
| > to finally take reproducibility more seriously
|
| I've long argued for this, as reproduction is the cornerstone
| of science. There's a lot of potential ways to do this but one
| that I like is linking to the original work. Suppose you're
| looking at the OpenReview page and they have a link for
| "reproduction efforts" and with at minimum an annotation for
| confirmation or failure.
|
| This is incredibly helpful to the community as a whole.
| Reproduction failures can be incredibly helpful even when the
| original work has no fraud. In those cases a reprising failure
| reveals important information about the necessary conditions
| that the original work relies on.
|
| But honestly, we'll never get this until we drop the entire
| notion of "novel" or "impact" and "publish or perish". Novel is
| in the eye of the reviewer and the lower the reviewer's
| expertise the less novel a work seems (nothing is novel as a
| high enough level). Impact can almost never be determined a
| priori, and when it can you already have people chasing those
| directions because why the fuck would they not? But publish or
| perish is the biggest sin. It's one of those ideas that looks
| nice on paper, like you are meaningfully determining who is
| working hard and who is hardly working. But the truth is that
| you can't tell without being in the weeds. The real result is
| that this stifles creativity, novelty, and impact as it forces
| researchers to chase lower hanging fruit. Things you're certain
| will work and can get published. It creates a negative feedback
| loop as we compete: "X publishes 5 papers a year, why can't
| you?" I've heard these words even when X has far fewer
| citations (each of my work had "more impact").
|
| Frankly, I believe fraud would dramatically reduce were
| researchers not risking job security. The fraud is incentivized
| by the cutthroat system where you're constantly trying to
| defend your job, your work, and your grants. They'll always be
| some fraud but (with a few exceptions) researchers aren't
| rockstar millionaires. It takes a lot of work to get to point
| where fraud even works, so there's a natural filter.
|
| I have the same advice as Mervin Kelly, former director of Bell
| Labs: How do you manage genius? You don't
| andai wrote:
| I heard that most papers in a given field are already not
| adding any value. (Maybe it depends on the field though.)
|
| There seems to be a rule in every field that "99% of everything
| is crap." I guess AI adds a few more nines to the end of that.
|
| The gems are lost in a sea of slop.
|
| So I see useless output (e.g. crap on the app store) as having
| negative value, because it takes up time and space and energy
| that could have been spent on something good.
|
| My point with all this is that it's not a new problem. It's
| always been about curation. But curation doesn't scale. It
| already didn't. I don't know what the answer to that looks
| like.
| qwertox wrote:
| It would be great if those scientists who use AI without
| disclosing it get fucked for life.
| direwolf20 wrote:
| "scientists" FYI. Making shit up isn't science.
| yesitcan wrote:
| One fuck seems appropriate.
| oofbey wrote:
| Harsh sentiment. Pretty soon every knowledge worker will use AI
| every day. Should people disclose spellcheckers powered by AI?
| Disclosing is not useful. Being careful in how you use it and
| checking work is what matters.
| ambicapter wrote:
| > Should people disclose spellcheckers powered by AI?
|
| Thank you for that perfect example of a strawman argument!
| No, spellcheckers that use AI is not the main concern behind
| disclosing the use of AI in generating scientific papers,
| government reports, or any large block of nonfiction text
| that you paid for that is supposed to make to sense.
| fisf wrote:
| People are accountable for the results they produce using AI.
| So a scientist is responsible for made up sources in their
| paper, which is plain fraud.
| oofbey wrote:
| I completely agree. But "disclosing the use of AI" doesn't
| solve that one bit.
| barbazoo wrote:
| I don't disclose what keyboard I use to write my code or
| if I applied spellcheck afterward. The result is 100%
| theirs.
| eichin wrote:
| "responsible for made up sources" leads to the hilarious
| idea that if you cite a paper that doesn't exist, you're
| now obliged to _write that paper_ (getting it retroactively
| published might be a challenge though)
| Proziam wrote:
| False equivalence. This isn't about "using AI" it's about
| having an AI pretend to do your job.
|
| What people are pissed about is the fact their tax dollars
| fund fake research. It's just fraud, pure and simple. And
| fraud should be punished brutally, especially in these cases,
| because the long tail of negative effects produces enormous
| damage.
| freedomben wrote:
| I was originally thinking you were being way too harsh with
| your "punish criminally" take, but I must admit, you're
| winning me over. I think we would need to be careful to
| ensure we never (or realistically, very rarely) convict an
| innocent person, but this is in many cases outright
| theft/fraud when someone is making money or being
| "compensated" for producing work that is fraudulent.
|
| For people who think this is too harsh, just remember we
| aren't talking about undergrads who cheat on a course paper
| here. We're talking about people who were given money
| (often from taxpayers) that committed fraud. This is
| textbook white collar crime, not some kid being lazy. At a
| _minimum_ we should be taking all that money back from them
| and barring them from ever receiving grant money again. In
| some cases I think fines exceeding the money they received
| would be appropriate.
| geremiiah wrote:
| What they are doing is plain cheating the system to get their
| 3 conference papers so they can get their $150k+ job at
| FAANG. It's plain cheating with no value.
| barbazoo wrote:
| People that cheat with AI now probably found ways to cheat
| before as well.
| shermantanktop wrote:
| Cheating by people in high status positions should get the
| hammer. But it gets the hand-wringing what-have-we-come-to
| treatment instead.
| WarmWash wrote:
| We are only looking at one side of the equation here, in
| this whole thread.
|
| This feels a bit like the "LED stoplights shouldn't be used
| because they don't melt snow" argument.
| mikkupikku wrote:
| Confront the culprit and ask for their side; you'll just
| get some sob story about how busy they are and how they
| were only using the AI to check their grammar and they
| just don't know how the whole thing ended up
| fabricated... Waste of time. Just blacklist these people,
| they're no better than any other scammer.
| Der_Einzige wrote:
| Rookie numbers. After NeurIPS main conference, you're dumb
| not to ask for 300K YOY. I watched IBM pay that amount
| prorated to an intern with a single first author NeurIPS
| publication.
| vimda wrote:
| "Pretty soon every knowledge worker will use AI every day" is
| a wild statement considering the reporting that most
| companies deploying AI solutions are seeing little to no
| benefit, but also, there's a pretty obvious gap between spell
| checkers and tools that generate large parts of the document
| for you
| PunchyHamster wrote:
| nice job moving the goalpost from "hallucinated the
| research/data" to "spellchecker error"
| duskdozer wrote:
| >Pretty soon every knowledge worker will use AI every day.
|
| Maybe? There's certainly a push to force the perception of
| inevitability.
| Sharlin wrote:
| In general we're pretty good at drawing a line between purely
| editorial stuff like using a spellchecker, or even the
| services a professional editor (no need to acknowledge), and
| independent intellectual contribution (must be acknowledged).
| There's no slippery slope.
| bwfan123 wrote:
| > It would be great if those scientists who use AI without
| disclosing it get fucked for life.
|
| There need to be dis-incentives for sloppy work. There is a
| tension between quality and quantity in almost every product.
| Unfortunately academia has become a numbers-game with paper-
| mills.
| pandemic_region wrote:
| Instead of publishing their papers in the prestigious zines -
| which is what they're after - we will publish them in "AI Slop
| Weekly" with name and picture. Up the submission risk a bit.
| jordanpg wrote:
| If these are so easy to identify, why not just incorporate some
| kind of screening into the early stages of peer review?
| DetectDefect wrote:
| Because real work takes time and effort, and there is no real
| incentive for it here.
| tossandthrow wrote:
| What makes you believe that are easy to identify?
| emil-lp wrote:
| One could require DOIs for each reference. That's both
| realistic to achieve and easy to verify.
|
| Although then why not just cite existing papers for bogus
| reasons?
| jordanpg wrote:
| Isn't that what GPTZero does?
| direwolf20 wrote:
| Wow! They're literally submitting references to papers by
| Firstname Lastname, John Doe and Jane Smith and nobody is
| noticing or punishing them.
| emil-lp wrote:
| They might (I hope) still be punished after discovery.
| an0malous wrote:
| It's the way of the future
| CGMthrowaway wrote:
| Which is worse:
|
| a) p-hacking and suppressing null results
|
| b) hallucinations
|
| c) falsifying data
|
| Would be cool to see an analysis of this
| Proziam wrote:
| All 3 of these should be categorized as fraud, and punished
| criminally.
| internetter wrote:
| criminally feels excessive?
| Proziam wrote:
| If I steal hundreds of thousands of dollars (salary, plus
| research grants and other funds) and produce fake output,
| what do you think is appropriate?
|
| To me, it's no different than stealing a car or tricking an
| old lady into handing over her fidelity account. You are
| stealing, and society says stealing is a criminal act.
| WarmWash wrote:
| We have a civil court system to handle stuff like this
| already.
| Proziam wrote:
| Stealing more than a few thousand dollars is a felony,
| and felonies are handled in criminal court, not civil.
|
| EDIT - The threshold amount varies. Sometimes it's as low
| as a few hundred dollars. However, the point stands on
| its own, because there's no universe where the sum in
| question is in misdemeanor territory.
| WarmWash wrote:
| It would fall under the domain of contract law, because
| maybe the contract of the grant doesn't prohibit what the
| researcher did. The way to determine that would be in
| court - civil court.
|
| Most institutions aren't very chill with grant money
| being misused, so we already don't need to burden then
| state with getting Johnny muncipal prosecutor to try and
| figure out if gamma crystallization imaging sources were
| incorrect.
| Proziam wrote:
| Fraud implies intent, either intent to deceive or
| intentionally negligent.
|
| If you're taking public funds (directly or otherwise)
| with the intent to either:
|
| A) Do little to no real work, and pass of the work of an
| AI as being your own work, or
|
| B) Knowingly publish falsified data
|
| Then you are, without a single shred of doubt, in
| criminal fraud territory. Further, the structural damage
| you inflict when you do the above is orders of magnitude
| greater than the initial fraud itself. That is a matter
| for civil courts ("Our company based on development on X
| fraudulent data, it cost us Y in damages").
|
| Whether or not charges are pressed is going to happen
| _way_ after all the internal reviews have demonstrated
| the person being charged has gone beyond the "honest
| mistake" threshold. It's like Walmart not bothering to
| call the cops until you're into felony territory, there's
| no point in doing so.
| wat10000 wrote:
| We also have a criminal court system to handle stuff like
| this.
| WarmWash wrote:
| No we don't. I've never seen a private contract dispute
| go to criminal court, probably because it's a civil
| matter.
|
| If they actually committed theft, well then that already
| is illegal too.
|
| But right now, doing "shitty research" isn't illegal and
| it's unlikely it ever will be.
| wat10000 wrote:
| The claim is that this would qualify as fraud, which is
| also illegal.
|
| If you do a search for "contractor imprisoned for fraud"
| you'll find plenty of cases where a private contract
| dispute resulted in criminal convictions for people who
| took money and then didn't do the work.
|
| I don't know if taking money and then merely pretending
| to do the research would rise to the level of criminal
| fraud, but it doesn't seem completely outlandish.
| jacquesm wrote:
| You could make a good case for a white collar crime here,
| fraud for instance.
| Der_Einzige wrote:
| Only when we can arrest people who say dumb stuff on the
| internet too. Much like how trump and bubba (bill Clinton)
| should share a jail cell, those who pontificate about what
| they don't know about (I.e non academics critiquing academia)
| can sit in the same jail cell as the supposed criminal
| academics.
|
| You gotta horse trade if you want to win. Take one for the
| team or get out of the way.
| amitav1 wrote:
| I'm doing some research, and this is something I'm unsure of. I
| see that "suppressing null results" is a bad thing, and I sort
| of agree, but for me personally, a lot of the null results are
| just the result of my own incompetence and don't contain any
| novel insights.
| fulafel wrote:
| Is there a comparison to rate of reference errors in other
| forums?
| dtartarotti wrote:
| It is very concerning that these hallucinations passed through
| peer review. It's not like peer review is a fool-proof method or
| anything, but the fact that reviewers did not check all
| references and noticed clearly bogus ones is alarming and could
| be a sign that the article authors weren't the only ones using
| LLMs in the process...
| amanaplanacanal wrote:
| Is it common for peer reviewers to check references? Somehow I
| thought they mostly focused on whether the experiment looked
| reasonable and the conclusions followed.
| emil-lp wrote:
| In journal publications it is, but without DOIs it's
| difficult.
|
| In conference publications, it's less common.
|
| Conference publications (like NEURips) is treated as
| announcement of results, not verified.
| empiko wrote:
| Nobody in ML or AI is verifying all your references.
| Reviewers will point out if you miss a super related work,
| but that's it. This is especially true with the recent
| (last two decades?) inflation in citation counts. You
| regularly have papers with 50+ references for all kinds of
| claims and random semirelated work. The citation culture is
| really uninspiring.
| smallpipe wrote:
| Could you run a similar analysis for pre-2020 papers? It'd be
| interesting to know how prevalent making up sources was before
| LLMs.
| tasuki wrote:
| Also, it'd be interesting how many pre-2020 papers their "AI
| detector" marks as AI-generated. I distrust LLMs somewhat, but
| I distrust AI detectors even more.
| theptip wrote:
| Yeah, it's kind of meaningless to attribute this to AI without
| measuring the base rate.
|
| It's for sure plausible that it's increasing, but I'm certain
| this kind of thing happened with humans too.
| bonsai_spool wrote:
| This suggests that nobody was screening this papers in the first
| place--so is it actually significant that people are using LLMs
| in a setting without meaningful oversight?
|
| These clearly aren't being peer-reviewed, so there's no natural
| check on LLM usage (which is different than what we see in work
| published in journals).
| emil-lp wrote:
| As one who reviews 20+ papers per year, we don't have time to
| verify each reference.
|
| We verify: is the stuff correct, and is it worthy of
| publication (in the given venue) given that it is correct.
|
| There is still some trust in the authors to not submit made-up-
| stuff, albeit it is diminishing.
| paulmist wrote:
| I'm surprised the conference doesn't provide tooling to
| validate all references automatically.
| Sharlin wrote:
| How would you do that? Even in cases where there's a
| standard format, a DOI on every reference, and some giant
| online library of publication metadata, including
| everything that only exists in dead tree format, that just
| lets you check whether the cited work exists, not whether
| it's actually a relevant thing to cite in the context.
| its_ethan wrote:
| Sorry, but if someone makes a claim and cites a reference,
| how do you verify "is the stuff correct" without checking
| that reference?
| emil-lp wrote:
| Those are typically things you are familiar with or can
| easily check.
|
| Fake references are more common in the introduction where
| you list relevant material to strengthen your results. They
| often don't change the validity of the claim, but the
| potential impact or value.
| gcr wrote:
| Academic venues don't have enough reviewers. This problem isn't
| new, and as publication volumes increase, it's getting sharply
| worse.
|
| Consider the unit economics. Suppose NeurIPS gets 20,000 papers
| in one year. Suppose each author should expect three good
| reviews, so area chairs assign five reviewers per paper. In
| total, 100,000 reviews need to be written. It's a lot of work,
| even before factoring emergency reviewers in.
|
| NeurIPS is one venue alongside CVPR, [IE]CCV, COLM, ICML,
| EMNLP, and so on. Not all of these conferences are as large as
| NeurIPS, but the field is smaller than you'd expect. I'd guess
| there are 300k-1m people in the world who are qualified to
| review AI papers.
| khuey wrote:
| Seems like using tooling like this to identify papers with
| fake citations and auto-rejecting them before they ever get
| in front of a reviewer would kill two birds with one stone.
| gcr wrote:
| It's not always possible to distinguish between fake
| citations and citations that are simply hard to find (e.g.
| wonderful old books that aren't on the Internet).
|
| Another problem is that conferences move slowly and it's
| hard to adjust the publication workflow in such an invasive
| way. CVPR only recently moved from Microsoft's CMT to
| OpenReview to accept author submissions, for example.
|
| There's a lot of opportunity for innovation in this space,
| but it's hard when everyone involved would need to agree to
| switch to a different workflow.
|
| (Not shooting you down. It's just complicated because the
| people who would benefit are far away from the people who
| would need to do the work to support it...)
| khuey wrote:
| Sure, I agree that it's far from trivial to implement.
| alain94040 wrote:
| When I was reviewing such papers, I didn't bother checking that
| 30+ citations were correctly indexed. I focused on the article
| itself, and maybe 1 or 2 citations that are important. That's
| it. For most citations, they are next to an argument that I
| know is correct, so why would I bother checking. What else do
| you expect? My job was to figure out if the article ideas are
| novel and interesting, not if they got all their citations
| right.
| geremiiah wrote:
| A lot of research in AI/ML seems to me to be "fake it and never
| make it". Literally it's all about optics, posturing,
| connections, publicity. Lots of bullshit and little substance.
| This was true before AI slop, too. But the fact that AI slop can
| make it pass the review really showcases how much a paper's
| acceptance hinges on things, other than the substance and results
| of the paper.
|
| I even know PIs who got fame and funding based on some research
| direction that supposedly is going to be revolutionary. Except
| all they had were preliminary results that from one angle, if you
| squint, you can envision some good result. But then the result
| never comes. That's why I say, "fake it, and never make it".
| gcr wrote:
| I was getting completely AI-generated reviews for a WACV
| publication back in 2024. The area chairs are so overworked that
| authors don't have much recourse, which sucks but is also really
| hard to handle unless more volunteers step up to the bat to help
| organize the conference.
|
| (If you're qualified to review papers, please email the program
| chair of your favorite conference and let them know -- they
| really need the help!)
|
| As for my review, the review form has a textbox for a summary, a
| textbox for strengths, a textbox for weaknesses, and a textbox
| for overall thoughts. The review I received included one complete
| set of summary/strengths/weaknesses/closing thoughts in the
| summary text box, another distinct set of
| summary/strengths/weaknesses/closing thoughts in the strengths,
| another complete and distinct review in the weaknesses, and a
| fourth complete review in the closing thoughts. Each of these
| four reviews were slightly different and contradicted each other.
|
| The reviewer put my paper down as a weak reject, but also said
| "the pros greatly outweigh the cons."
|
| They listed "innovative use of synthetic data" as a strength, and
| "reliance on synthetic data" as a weakness.
| cubefox wrote:
| Wow...
| Tom1380 wrote:
| No ETH Zurich, let's go
| gcr wrote:
| NeurIPS leadership doesn't think hallucinated references are
| necessarily disqualifying; see the full article from Fortune for
| a statement from them: https://archive.ph/yizHN
|
| > When reached for comment, the NeurIPS board shared the
| following statement: "The usage of LLMs in papers at AI
| conferences is rapidly evolving, and NeurIPS is actively
| monitoring developments. In previous years, we piloted policies
| regarding the use of LLMs, and in 2025, reviewers were instructed
| to flag hallucinations. Regarding the findings of this specific
| work, we emphasize that significantly more effort is required to
| determine the implications. Even if 1.1% of the papers have one
| or more incorrect references due to the use of LLMs, the content
| of the papers themselves are not necessarily invalidated. For
| example, authors may have given an LLM a partial description of a
| citation and asked the LLM to produce bibtex (a formatted
| reference). As always, NeurIPS is committed to evolving the
| review and authorship process to best ensure scientific rigor and
| to identify ways that LLMs can be used to enhance author and
| reviewer capabilities."
| Analemma_ wrote:
| Kinda gives the whole game away, doesn't it? "It doesn't
| actually matter if the citations are hallucinated."
|
| In fairness, NeurIPS is just saying out loud what everyone
| already knows. Most citations in published science are useless
| junk: it's either mutual back-scratching to juice h-index, or
| it's the embedded and pointless practice of overcitation, like
| "Human beings need clean water to survive (Franz, 2002)".
|
| Really, hallucinated citations are just forcing a reckoning
| which has been overdue for a while now.
| jacquesm wrote:
| There should be a way to drop any kind of circular citation
| ring from the indexes.
| gcr wrote:
| It's tough because some great citations are hard to
| find/procure still. I sometimes refer to papers that aren't
| on the Internet (eg. old wonderful books / journals).
| jacquesm wrote:
| But that actually strengthens those citiations. The I
| scratch your back you scratch mine ones are the ones I'm
| getting at and that is quite hard to do with old and
| wonderful stuff, the authors there are probably not in a
| position to reciprocate by virtue of observing the grass
| from the other side.
| gcr wrote:
| I think it's a hard problem. The semanticscholar folks
| are doing the sort of work that would allow them to track
| this; I wonder if they've thought about it.
|
| A somewhat-related parable: I once worked in a larger lab
| with several subteams submitting to the same conference.
| Sometimes the work we did was related, so we both cited
| each other's paper which was also under review at the
| same venue. (These were flavor citations in the "related
| work" section for completeness, not material to our
| arguments.) In the review copy, the reference lists the
| other paper as written by "anonymous (also under review
| at XXXX2025)," also emphasized by a footnote to explain
| the situation to reviewers. When it came time to submit
| the camera-ready copy, we either removed the
| anonymization or replaced it with an arxiv link if the
| other team's paper got rejected. :-) I doubt this
| practice improved either paper's chances of getting
| accepted.
|
| Are these the sorts of citation rings you're talking
| about? If authors misrepresented the work as if it were
| accepted, or pretended it was published last year or
| something, I'd agree with you, but it's not too uncommon
| in my area for well-connected authors to cite manuscripts
| in process. I don't think it's a problem as long as they
| don't lean on them.
| jacquesm wrote:
| No, I'm talking about the ones where the citation itself
| is almost or even completely irrelevant and used as a way
| to inflate the citation count of the authors. You could
| find those by checking whether or not the value as a
| reference (ie: contributes to the understanding of the
| paper you are reading) is exceeded by the value of the
| linkage itself.
| zipy124 wrote:
| The flavour citations in related work is the best place
| to launder citations.
| fc417fc802 wrote:
| > Most citations in published science are useless junk:
|
| Can't say that matches my experience at all. Once I've found
| a useful paper on a topic thereafter I primarily navigate the
| literature by traveling up and down the citation graph. It's
| extremely effective in practice and it's continued to get
| easier to do as the digitization of metadata has improved
| over the years.
| empath75 wrote:
| I think a _single_ instance of an LLM hallucination should be
| enough to retract the whole paper and ban further submissions.
| gcr wrote:
| Going through a retraction and blacklisting process is also a
| lot of work -- collecting evidence, giving authors a chance
| to respond and mediate discussion, etc.
|
| Labor is the bottleneck. There aren't enough academics who
| volunteer to help organize conferences.
|
| (If a reader of this comment is qualified to review papers
| and wants to step up to the plate and help do some work in
| this area, please email the program chairs of your favorite
| conference and let them know. They'll eagerly put you to
| work.)
| pessimizer wrote:
| That's exactly why the inclusion of a hallucinated
| reference is actually a blessing. Instead going back and
| forth with the fraudster, just tell them to find the paper.
| If they can't, case closed. Massive amount of time and
| money saved.
| gcr wrote:
| Isn't telling them to find the paper just "going back and
| forth with a fraudster"?
|
| One "simple" way of doing this would be to automate it.
| Have authors step through a lint step when their camera-
| ready paper is uploaded. Authors would be asked to
| confirm each reference and link it to a google scholar
| citation. Maybe the easy references could be auto-
| populated. Non-public references could be resolved by
| uploading a signed statement or something.
|
| There's no current way of using this metadata, but it
| could be nice for future systems.
|
| Even the Scholar team within Google is woefully
| understaffed.
|
| My gut tells me that it's probably more efficient to just
| drag authors who do this into some public execution or
| twitter mob after-the-fact. CVPR does this every so often
| for authors who submit the same paper to multiple venues.
| You don't need a lot of samples for deterrence to take
| effect. That's kind of what this article is doing, in a
| sense.
| wing-_-nuts wrote:
| I dunno about _banning_ them, humans without LLMs make
| mistakes all the time, but I would definitely place them
| under much harder scrutiny in the future.
| pessimizer wrote:
| Hallucinations aren't mistakes, they're fabrications. The
| two are probably referred to by the same word in some
| languages.
|
| Institutions can choose an arbitrary approach to mistakes;
| maybe they don't mind a lot of them because they want to
| take risks and be on the bleeding edge. But any flexible
| attitude towards fabrications is simply corruption. The
| connected in-crowd will get mercy and the outgroup will get
| the hammer. Anybody criticizing the differential treatment
| will be accused of supporting the outgroup fraudsters.
| gcr wrote:
| Fabrications carry intent to decieve. I don't think
| hallucinations necessarily do. If anything, they're a
| matter of negligence, not deception.
|
| Think of it this way: if I wanted to commit pure academic
| fraud maliciously, I wouldn't make up a fake reference.
| Instead, I'd find an existing related paper and merely
| misrepresent it to support my own claims. That way, the
| deception is much harder to discover and I'd have
| plausible deniability -- "oh I just misunderstood what
| they were saying."
|
| I think most academic fraud happens in the figures, not
| the citations. Researchers are more likely to to be
| successful at making up data points than making up
| references because it's impossible to know without the
| data files.
| direwolf20 wrote:
| Generating a paper with an LLM is already academic fraud.
| You, the fraudster, are trying to optimize your fraud-to-
| effort ratio which is why you don't bother to look for
| existing papers to mis-cite.
| andy99 wrote:
| For example, authors may have given an LLM a partial
| description of a citation and asked the LLM to produce bibtex
|
| This is equivalent to a typo. I'd like to know which
| "hallucinations" are completely made up, and which have a
| corresponding paper but contain some error in how it's cited.
| The latter I don't think matters.
| burkaman wrote:
| If you click on the article you can see a full list of the
| hallucinations they found. They did put in the effort to
| look for plausible partial matches, but most of them are
| some variation of "No author or title match. Doesn't exist
| in publication."
|
| Here's a random one I picked as an example.
|
| Paper: https://openreview.net/pdf?id=IiEtQPGVyV
|
| Reference: Asma Issa, George Mohler, and John Johnson.
| Paraphrase identification using deep contextual- ized
| representations. In Proceedings of the 2018 Conference on
| Empirical Methods in Natural Language Processing (EMNLP),
| pp. 517-526, 2018.
|
| Asma Issa and John Johnson don't appear to exist. George
| Mohler does, but it doesn't look like he works in this area
| (https://www.georgemohler.com/). No paper with that title
| exists. There are some with sort of similar titles
| (https://arxiv.org/html/2212.06933v2 for example), but none
| that really make sense as a citation in this context. EMNLP
| 2018 exists (https://aclanthology.org/D18-1.pdf), but that
| page range is not a single paper. There are papers in there
| that contain the phrases "paraphrase identification" and
| "deep contextualized representations", so you can see how
| an LLM might have come up with this title.
| gold23 wrote:
| It's not the equivalent of a typo. A typo would be
| immediately apparent to the reader. This is a semantic
| error that is much less likely to be caught by the reader.
| jklinger410 wrote:
| > the content of the papers themselves are not necessarily
| invalidated. For example, authors may have given an LLM a
| partial description of a citation and asked the LLM to produce
| bibtex (a formatted reference)
|
| Maybe I'm overreacting, but this feels like an insanely biased
| response. They found the one potentially innocuous reason and
| latched onto that as a way to hand-wave the entire problem
| away.
|
| Science already had a reproducibility problem, and it now has a
| hallucination problem. Considering the massive influence the
| private sector has on the both the work and the institutions
| themselves, the future of open science is looking bleak.
| paulmist wrote:
| Isn't disqualifying X months of potentially great research
| due to a misformed, but existing reference harsh? I don't
| think they'd be okay with references that are actually made
| up.
| suddenlybananas wrote:
| It's a sign of dishonesty, not a perfect one, but an
| indicator.
| jklinger410 wrote:
| When your entire job is confirming that science is valid, I
| expect a little more humility when it turns out you've
| missed a critical aspect.
|
| How did these 100 sources even get through the validation
| process?
|
| > Isn't disqualifying X months of potentially great
| research due to a misformed, but existing reference harsh?
|
| It will serve as a reminder not to cut any corners.
| paulmist wrote:
| > When your entire job is confirming that science is
| valid, I expect a little more humility when it turns out
| you've missed a critical aspect.
|
| I wouldn't call a misformed reference a critical issue,
| it happens. That's why we have peer reviews. I would
| contend drawing superficially valid conclusions from
| studies through use of AI is a much more burning problem
| that speaks more to the integrity of the author.
|
| > It will serve as a reminder not to cut any corners.
|
| Or yet another reason to ditch academic work for
| industry. I doubt the rise of scientific AI tools like
| AlphaXiv [1], whether you consider them beneficial or
| detrimental, can be avoided - calling for a level
| pragmatism.
| jklinger410 wrote:
| > I wouldn't call a misformed reference a critical issue,
| it happens. That's why we have peer reviews.
|
| Crazy to say this in a discussion where peer review
| missed hallucinated citations
| pas wrote:
| even the fact that citations are not automatically
| verified by the journal is crazy, the whole academia and
| publishing enterprise is an empire built on inefficiency,
| hubris, and politics (but I'm repeating myself).
| zipy124 wrote:
| Science relies on trust.. a lot. So things which show
| dishonesty are penalised greatly. If we were to remove
| trust then peer reviewing a paper might take months of work
| or even years.
| paulmist wrote:
| And that timeline only grows with the complexity of the
| field in question. I think this is inherently a function
| of the complexity of the study, and rather than harshly
| penalizing such shortcomings we should develop tools that
| address them and improve productivity. AI can speed up
| the verification of requirements like proper citations,
| both on the author's and reviewer's side.
| loglog wrote:
| Math does that. Peer review cycles are measured in years
| there. This does not stop fashionable subfields from
| publishing sloppy papers, and occasionally even
| irrecoverably false ones.
| orbital-decay wrote:
| The wording is not hand-wavy. They said "not necessarily
| invalidated", which could mean that innocuous reason and
| nothing extra.
| mikkupikku wrote:
| Even if some of those innocuous mistakes happen, we'll all
| be better off if we accept people making those mistakes as
| acceptable casualties in an unforgiving campaign against
| academic fraudsters.
|
| It's like arguing against strict liability for drunk
| driving because maybe somebody accidentally let their grape
| juice sit to long and they didn't know it was fermented...
| I can conceive of such a thing, but that doesn't mean we
| should go easy on drunk driving.
| jklinger410 wrote:
| I really think it is. The primary function of these
| publications is to validate science. When we find invalid
| citations, it shows they're not doing their job. When they
| get called on that, they cite the volume of work their
| publication puts out and call out the only potential not-
| disqualifying outcome.
|
| Seems like CYA, seems like hand wave. Seems like excuses.
| derf_ wrote:
| This will continue to happen as long as it is effectively
| unpunished. Even retracting the paper would do little good, as
| odds are it would not have been written if the author could not
| have used an LLM, so they are no worse off for having tried.
| Scientific publications are mostly a numbers game at this
| point. It is just one more example of a situation where
| behaving badly is much cheaper than policing bad behavior, and
| until incentives are changed to account for that, it will only
| get worse.
| Aurornis wrote:
| > Even if 1.1% of the papers have one or more incorrect
| references due to the use of LLMs, the content of the papers
| themselves are not necessarily invalidated.
|
| This statement isn't wrong, as the rest of the paper could
| still be correct.
|
| However, when I see a blatant falsification somewhere in a
| paper I'm immediately suspicious of everything else. Authors
| who take lazy shortcuts when convenient usually don't just do
| it once, they do it wherever they think they can get away with
| it. It's a slippery slope from letting an LLM handle citations
| to letting the LLM write things for you to letting the LLM
| interpret the data. The latter opens the door to hallucinated
| results and statistics, as anyone who has experimented with
| LLMs for data analysis will discover eventually.
| mlmonkey wrote:
| Why not run every submitted paper through GPTZero (before
| sending to reviewers) and summarily reject any paper with a
| hallucination?
| gcr wrote:
| That's how GPTZero wants to situate themselves.
|
| Who would pay them? Conference organizers are already unpaid
| and undestaffed, and most conferences aren't profitable.
|
| I think rejections shouldn't be automatic. Sometimes there
| are just typos. Sometimes authors don't understand BibTeX.
| This needs to be done in a way that reduces the workload for
| reviewers.
|
| One way of doing this would be for GPTZero to annotate each
| paper during the review step. If reviewers could review a
| version of each paper with yellow-highlighted "likely-
| hallucinated" references in the bibliography, then they'd
| bring it up in their review and they'd know to be on their
| guard for other probably LLM-isms. If there's only a couple
| likely typos in the references, then reviewers could
| understand that, and if they care about it, they'd bring it
| up in their reviews and the author would have the usual
| opportunity to rebut.
|
| I don't know if GPTZero is willing to provide this service
| "for free" to the academic community, but if they are, it's
| probably worth bringing up at the next PAMI-TC meeting for
| CVPR.
| zipy124 wrote:
| Most publication venues already pay for a plagiarism
| detection service, it seems it would be trivial to add it
| on as a cost. Especially given APCs for journals are
| several thousand dollars, what's a few dollars more per
| paper.
| Molitor5901 wrote:
| AI might just extinguish the entire paradigm of publish or
| perish. The sheer volume of papers makes it nearly impossible to
| properly decide which papers have merit, which are non-replicate
| and suspect, and which are just a desperate rush to publish. The
| entire practice needs to end.
| shermantanktop wrote:
| But how could we possibly evaluate faculty and researcher
| quality without counting widgets on an assembly line? /s
|
| It's a problem. The previous regime prior to publishing-mania
| was essentially a clubby game of reputation amongst peers based
| on cocktail party socialization.
|
| The publication metrics came out of the harder sciences, I
| believe, and then spread to the softest of humanities. It was
| always easy to game a bit if you wanted to try, but now it's
| trivial to defeat.
| SJC_Hacker wrote:
| Its not publish or perish so much as get grant money or perish.
|
| Publishing is just the way to get grants.
|
| A PI explained it to me once, something like this
|
| Idea(s) -> Grant -> Experiments -> Data -> Paper(s) ->
| Publication(s) -> Idea(s) -> Grant(s)
|
| Thats the current cycle ... remove any step and its a dead end
| armcat wrote:
| This is awful but hardly surprising. Someone mentioned
| reproducible code with the papers - but there is a high
| likelihood of the code being partially or fully AI generated as
| well. I.e. AI generated hypothesis -> AI produces code to
| implement and execute the hypothesis -> AI generates paper based
| on the hypothesis and the code.
|
| Also: there were 15 000 submissions that were rejected at
| NeurIPS; it would be very interesting to see what % of those
| rejected were partially or fully AI generated/hallucinated. Are
| the ratios comperable?
| blackbear_ wrote:
| Whether the code is AI generated or not is not important, what
| matters is that it really works.
|
| Sharing code enables others to validate the method on a
| different dataset.
|
| Even before LLMs came around there were lots of methods that
| looked good on paper but turned out not to work outside of
| accepted benchmarks
| depressionalt wrote:
| This is nice and all, but what repercussion does GPTZero get when
| their bullshit AI detection hallucinates a student using AI? And
| when that student receives academic discipline because of it?
|
| Many such cases of this. More than 100!
|
| They claim to have custom detection for GPT-5, Gemini, and
| Claude. They're making that up!
| freedomben wrote:
| Indeed. My son has been accused by bullshit AI detection as
| having used AI, and it has devastated his work quality. After
| being "disciplined" for using AI (when he didn't), he now
| intentionally tries to "dumb down" his writing so that it
| doesn't sound so much like AI. The result is he writes much
| worse. What a shitty, shitty outcome. I've even found myself
| leaving typos and things in (even on sites like HN) because if
| you write _too_ well, inevitably some comment replier will call
| you out as being an LLM even when you aren 't. I'm as annoyed
| by the LLM posts as everybody else, but the answer surely is
| not to dumb us down into Idiocracy.
| Sharlin wrote:
| It's almost as if this whole LLM stuff wasn't a net benefit
| to the society after all.
| Der_Einzige wrote:
| Stop using em dashes, the fancy quotes that can't be easily
| typed. Stop using overused words like certainly and delve.
| Stop using LLM template slop like "it's not X, it's Y". Stop
| always doing lists of 3s. We know you didn't use to use so
| many emojis or bolded text. Also, AI really fking hates the
| exclamation mark so that's a great proof of humanity!
|
| Most people getting flagged are getting flagged because they
| actually used AI and couldn't even be bothered to manually
| deslop it.
|
| People who are too lazy to put even a tiny bit of human
| intentionality into their work deserve it.
| theptip wrote:
| This is mostly an ad for their product. But I bet you can get
| pretty good results with a Claude Code agent using a couple
| simple skills.
|
| Should be extremely easy for AI to successfully detect
| hallucinated references as they are semi-structured data with an
| easily verifiable ground truth.
| leggerss wrote:
| I don't understand: why aren't there automated tools to verify
| citations' existence? The data for a citation has a structured
| styling (APA, MLA, Chicago) and paper metadata is available via
| e.g. a web search, even if the paper contents are not
|
| I guess GPTZero has such a tool. I'm confused why it isn't used
| more widely by paper authors and reviewers
| gh02t wrote:
| Citations are too open ended and prone to variation, and
| legitimate minor mistskes that wouldn't bother a human verifier
| but would break automated tools to easily verify in their
| current form. DOI was supposed to solve some of the literal
| mechanical variation of the existence of a source, but journal
| paywalls and limited adoption mean that is not a universal
| solution. Plus DOI still doesn't easily verify the factual
| accuracy of a citation, like "does the source say what the
| citation says it does," which is the most important part.
|
| In my experience you will see considerable variation in
| citation formats, even in journals that strictly define it and
| require using BibTex. And lots of journals leave their citation
| format rules very vague. Its a problem that runs deep.
| leggerss wrote:
| Thanks for the thoughtful reply!
| eichin wrote:
| Looks like GPTZero Source Finder was only released a year ago -
| if anything, I'm surprised slop-writers aren't using it
| preemptively, since they're "ahead of the curve" relative to
| reviewers on this sort of thing...
| yepyeaisntityea wrote:
| No surprises. Machine learning has, at least since 2012, been the
| go-to field for scammers and grifters. Machine learning, and
| technology in general, is basically a few real ideas, a small
| number of honest hard workers, and then millions of fad chasers
| and scammers.
| mt_ wrote:
| It would be ironic if the very detection of hallucinations
| contained hallucinations of its own.
| doug_durham wrote:
| Getting papers published is now more about embellishing your CV
| versus a sincere desire to present new research. I see this
| everywhere at every level. Getting a paper published _anywhere_
| is a checkbox in completing your resume. As an industry we need
| to stop taking this into consideration when reviewing candidates
| or deciding pay. In some sense it has become an anti-signal.
| londons_explore wrote:
| I'd like to see a financial approach to deciding pay by giving
| researchers a small and perhaps nonlinear or time bounded share
| of any profits that arise from their research.
|
| Then peoples CV's could say "My inventions have led to $1M in
| licensing revenue" rather than "I presented a useless idea at a
| decent conference because I managed to make it sound exciting
| enough to get accepted".
| direwolf20 wrote:
| That's what patents do.
| autoexec wrote:
| A lot of good research isn't ever going to make anyone a
| single dime, but that doesn't mean it doesn't matter.
| autoexec wrote:
| It'd be nice if there were a dedicated journal for papers
| published just because you have to publish for your CV or to
| get your degree. That way people can keep publishing for the
| sake of publishing, but you could see at a glance what the deal
| was.
| biophysboy wrote:
| I think its fairer to say that perverse incentives have added
| more noise to the publishing signal. Publishing 0 times is not
| better than 100 times, even if 90% of those are Nth author
| formality/politeness citations.
| nerdjon wrote:
| The downstream effects of this are extremely concerning. We have
| already seen the damage caused by human written research that was
| later retracted like the "research" on vaccines causing autism.
|
| As we get more and more papers that may be citing information
| that was originally hallucinated in the first place we have a
| major reliability issue here. What is worse is people that did
| not use AI in the first place will be caught in the crosshairs
| since they will be referencing incorrect information.
|
| There needs to be a serious amount of education done on what
| these tools can and cannot do and importantly where they fail.
| Too many people see these tools as magic since that is what the
| big companies are pushing them as.
|
| Other than that we need to put in actual repercussions for
| publishing work created by an LLM without validating it (or just
| say you can't in the first place but I guess that ship has
| sailed) or it will just keep happening. We can't just ignore it
| and hope it won't be a problem.
|
| And yes, humans can make mistakes too. The difference is
| accountability and the ability to actually be unsure about
| something so you question yourself to validate.
| pandemic_region wrote:
| What if they would only accept handwritten papers? Basically the
| current system is beyond repair, so may as well go back to
| receiving 20 decent papers instead of 20k hallucinated ones.
| ctoth wrote:
| How you know it's really real is that they clearly tell the FPR,
| and compare against a pre-llm baseline.
|
| But I saw it in Apple News, so MISSION ACCOMPLISHED!
| yobbo wrote:
| As long as these sorts of papers serve more important purposes
| for the careers of the authors than anything related to science
| or discovery of knowledge, then of course this happens and
| continues.
|
| The best possible outcome is that these two purposes are
| disconflated, with follow-on consequences for the conferences and
| journals.
| poulpy123 wrote:
| All papers proved to have used a LLM beyond writing improvement
| should be automatically retracted
| brador wrote:
| The problem isn't scale.
|
| The problem is consequences (lack of).
|
| Doing this should get you barred from research. It won't.
| CrzyLngPwd wrote:
| This is not the AI future we dreamed of, or feared.
| nospice wrote:
| We've been talking about a "crisis of reproducibility" for years
| and the incentive to crank out high volumes of low-quality
| research. We now have a tool that brings down the cost of
| producing plausibly-looking research down to zero. So of course
| we're going to see that tool abused on a galactic scale.
|
| But here's the thing: let's say you're an university or a
| research institution that wants to curtail it. You catch someone
| producing LLM slop, and you confirm it by analyzing their work
| and conducting internal interviews. You fire them. The fired
| researcher goes public saying that they were doing nothing of the
| sort and that this is a witch hunt. Their blog post makes it to
| the front page of HN, garnering tons of sympathy and prompting
| many angry calls to their ex-employer. It gets picked up by some
| mainstream outlets, too. It happened a bunch of times.
|
| In contrast, there are basically no consequences to institutions
| that let it slide. No one is angrily calling the employers of the
| authors of these 100 NeurIPS papers, right? If anything, there's
| the plausible deniability of "oh, I only asked ChatGPT to
| reformat the citations, the rest of the paper is 100% legit, my
| bad".
| meindnoch wrote:
| Jamie, bring up their nationalities.
| neom wrote:
| I wrote before about my embarrassing time with ChatGPT during a
| period (https://news.ycombinator.com/item?id=44767601) - I
| decided to go back through those old 4o chats with 5.2 pro
| extended thinking, the reply was pretty funny because it first
| slightly ridiculed me, heh - but what it showed was: basically I
| would say "what 5 research papers from any area of science talk
| to these ideas" and it would find 1 and invent 4 if it didn't
| know 4 others, and not tell me, and then I'd keep working with it
| and it would invent what it thought might be in the papers long
| the way, making up new papers in it's own work to cite to make
| it's own work valid, lol. Anyway, I'm a moron, sure, and no real
| harm came of it for _me_ , just still slightly shook I let that
| happen to me.
| Shocka1 wrote:
| Just to clarify, you didn't actually look up the publications
| it was citing? For example, you just stayed in ChatGPT web and
| used the resources it provided there? Not ridiculing you of
| course, but am just curious. The last paper I wrote a couple
| months back I had GPT search out the publications for me, but I
| would always open a new tab and retrieve the actual
| publication.
| neom wrote:
| I didn't because I wasn't really doing anything serious to my
| mind, I think? basically felt like watching an episode of pbs
| spacetime, I think the difference is it's more like playing a
| video game while thinking you're watching an episode of
| spacetime, if that makes sense? I don't use chatgpt for me
| real work that much, and I'm not a scientist, so it was for
| me just mucking around, it pushed me slightly over a line
| into "I was just playing but now this seems real", it didn't
| occur to me to go back through and check all the papers, I
| guess because quite a lot of chatting had happened since then
| and, I dunno, I just didn't think to? Not sure that makes
| much sense. This was also over a year ago, during the time
| they had the gpt4o sycophancy mode that made the news, and it
| wasn't backed by webserch, so I took for granted what was in
| it's training data. No good excuse I'm afraid. tldr: poor
| critical thinking skills on my part there! :)
| londons_explore wrote:
| And this is the tip of the iceberg, because these are the easy to
| check/validate things.
|
| I'm sure plenty of more nuanced facts are also entirely without
| basis.
| techIA wrote:
| They will turn it into a party drug.
| trash_cat wrote:
| Clearly there is some demand for those papers, and research, to
| exist. Good opportunity to fill the gaps.
| captainbland wrote:
| What's wild is so many of these are from prestigious
| universities. MIT, Princeton, Oxford and Cambridge are all on
| there. It must be a terrible time to be an academic who's getting
| outcompeted by this slop because somebody from an institution
| with a better name submitted it.
| cflewis wrote:
| I'm going to be charitable and say that the papers from
| prestigious universities were honest mistakes rather than paper
| mill university fabrications.
|
| One thing that has bothered me for a very long time is that
| computer science (and I assume other scientific fields) has
| long since decided that English is the lingua franca, and if
| you don't speak it you can't be part of it. Can you imagine if
| being told that you could only do your research if you were
| able to write technical papers in a language you didn't speak,
| maybe even using glyphs you didn't know? It's crazy when you
| think about it even a little bit, but we ask it of so many.
| Let's not include the fact that 90% of the English-speaking
| population couldn't crank out a paper to the required
| vocabulary level anyway.
|
| A very legitimate, not trying to cheat, use for LLMs is
| translation. While it would be an extremely broad and dangerous
| brush to paint with, I wonder if there is a correlation between
| English-as-a-Second (or even third)-Language authors and the
| hallucinations. That would indicate that they were trying to
| use LLMs to help craft the paper to the expected writing level.
| The only problem being that it sometimes mangles citations, and
| if you've done good work and got 25+ citations, it's easy for
| those errors to slip through.
| teekert wrote:
| We have the h score and such, can we have something similar that
| goes down when you pull stunts like these? Preferably link it to
| people's orcid ids.
| dev_l1x_be wrote:
| I am wondering if we are going to reach hallucination collapse
| sooner than we reach AGI.
| Nevermark wrote:
| With regard to confabulating (hallucinating) sources, or anything
| else, it is worth noting this is a first class training
| requirement imposed on models. Not models simply picking up the
| habit from humans.
|
| When training a student, normally we expect a lack of knowledge
| early, and reward self-awareness, self-evaluation and self-
| disclosure of that.
|
| But the very first epoch of a model training run, when the model
| has all the ignorance of a dropped plate of spaghetti, we
| optimize the network to respond to information, as anything from
| a typical human to an expert, without any base of understanding.
|
| So the training practice for models is inherently extreme
| enforced "fake it until you make it", to a degree far beyond any
| human context or culture.
|
| (Regardless, humans need to verify, not to mention read, the
| sources they site. But it will be nice when models can be trusted
| to accurately access what they know/don't-know too.)
| rabbitlord wrote:
| You will find out that Top CS conference is never scientific, if
| you really go to their GitHub and run their code.
| ctoth wrote:
| The innumeracy is load-bearing for the entire media ecosystem. If
| readers could do basic proportional reasoning, half of health
| journalism and most tech panic coverage would collapse overnight.
|
| GPTZero of course knows this. "100 hallucinations across 53
| papers at prestigious conference" hits different than "0.07% of
| citations had issues, compared to unknown baseline, in papers
| whose actual findings remain valid."
| MeetingsBrowser wrote:
| I'm not sure that's fair in this context.
|
| In the past, a single paper with questionable or falsified
| results at a top tier conference was big news.
|
| Something that casts doubt on the validity of 53 papers at a
| top AI conference is at least notable.
|
| > whose actual findings remain valid
|
| Remain valid according to who? The same group that missed
| hundreds of hallucinated citations?
| ctoth wrote:
| Which of these papers had falsified _results_ and not bad
| citations?
|
| What is the base rate of bad citations pre-AI?
|
| And finally yes. Peer review does not mean clicking every
| link in the footnotes to make sure the original paper didn't
| mislink, though I'm sure after this bruhaha this too will be
| automated.
| MeetingsBrowser wrote:
| > Peer review does not mean clicking every link in the
| footnotes
|
| It wasn't just broken links, but citing authors like
| "lastname, firstname" and made up titles.
|
| I have done peer reviews for a (non-AI) CS conference and
| did at least skim the citations. For papers related to my
| domain, I was familiar with most of the citations already,
| and looked into any that looked odd.
|
| Being familiar with the state of the art is, in theory,
| what qualifies you to do peer reviews.
| rfrey wrote:
| There's a lot of good arguments in this thread about incentives:
| extremely convincing about why current incentives lead to exactly
| this behaviour, and also why creating better incentives is a very
| hard problem.
|
| If we grant that good carrots are hard to grow, what's the
| argument against leaning into the stick? Change university
| policies and processes so that getting caught fabricating data or
| submitting a paper with LLM hallucinations is a career ending
| event. Tip the expected value of unethical behaviours in favour
| of avoiding them. Maybe we can't change the odds of getting
| caught but we certainly can change the impact.
|
| This would not be easy, but maybe it's more tractable than
| changing positive incentives.
| currymj wrote:
| the harsher the punishment, the more due process required.
|
| i don't think there are any AI detection tools that are
| sufficiently reliable that I would feel comfortable expelling a
| student or ending someone's career based on their output.
|
| for example, we can all see what's going on with these papers
| (and it appears to be even worse among ICLR submissions). but
| it is possible to make an honest mistake with your BibTeX. Or
| to use AI for grammar editing, which is widely accepted, and
| have it accidentally modify a data point or citation. There are
| many innocent mistakes which also count as plausible excuses.
|
| in some cases further investigation maybe can reveal a smoking
| gun like fabricated data, which is academic misconduct whether
| done by hand or because an AI generated the LaTeX tables.
| punishments should be harsher for this than they are.
| rfrey wrote:
| Fabricated citations seem to be a popular and non ambiguous
| way for AI to sabotage science.
| godelski wrote:
| Given that many of these detections are being made from
| references, I don't understand why we're not using automatic
| citation checkers.
|
| Just ask authors to submit their bib file so we don't need to do
| OCR on the PDF. Flag the unknown citations and ask reviewers to
| verify their existence. Then contact authors and ban if they
| can't produce the cited work.
|
| This is low hanging fruit here!
|
| Detecting slop where the authors vet citations is much harder.
| The big problem with all the review rules is they have no teeth.
| If it were up to me we'd review in the open, or at least like
| ICLR. Publish the list of known bad actors and let is look at the
| network. The current system is too protective of egregious errors
| like plagiarism. Authors can get detected in one conference,
| pull, and submit to another, rolling the dice. We can't allow
| that to happen and we should discourage people from associating
| with these conartists.
|
| AI is certainly a problem in the world of science review, but
| it's far from the only one and I'm not even convinced it's the
| biggest. The biggest is just that reviewers are lazy and/or not
| qualified to review the works they're assigned. It takes at least
| an hour to properly review a paper in your niche, much more when
| it's outside. We're over worked as is, with 5+ works to review,
| not to mention all the time we got to spend reworking our own
| works that were rejected due to the slot machine. We could do
| much better if we dropped this notion of conference/journal
| prestige and focused on the quality of the works and reviews.
|
| Addressing those issues also addresses the AI issues because,
| frankly, * _it doesn 't matter if the whole work was done by AI,
| what matters is if the work is real.*_
| pacbard wrote:
| The ironic part about these hallucinations is that a research
| paper includes a literature review because the goal of the
| research is to be in dialogue with prior work, to show a gap in
| the existing literature, and to further the knowledge that this
| prior work has built.
|
| By using an LLM to fabricate citations, authors are moving away
| from this noble pursuit of knowledge built on the "shoulders of
| giants" and show that behind the curtain output volume is what
| really matters in modern US research communities.
| andy_xor_andrew wrote:
| I guess that makes this "standing on the shoulders of
| fabrications"
| stogot wrote:
| Fabrication should be immediate academic ban for life
| physPop wrote:
| Paper mills existed long before LLMs
| AlienRobot wrote:
| That's going to be the philosophical question of our times: do
| LLMs generate slop out of nowhere or does it simply amplify the
| slop machinery that was already there?
| abktowa wrote:
| Implicitly this makes sense but the amount cited in this article
| is still hard for me to grasp. Wow.
| gtirloni wrote:
| Why focus on hallucinations/LLMs and not on the authors? There
| are rules for submitting papers.
|
| If I drop a loaded gun and it fires, killing someone, we don't go
| after the gun's manufacturer in most cases.
| phyzome wrote:
| This isn't directly to your point, but: A civil suit for such
| an incident would generally name both the weapon owner (for
| negligence, etc.) and the manufacturer (for dangerous design).
| Der_Einzige wrote:
| Actually, if you're the US navy, you DO go after the
| manufacturer!
|
| Go look up the P320 pistol and the tons of accidental
| discharges that's it's caused.
|
| https://stateline.org/2025/03/10/more-law-enforcement-agenci...
| not2b wrote:
| This is going to be a huge problem for conferences. While
| journals have a longer time to get things right, as a conference
| reviewer (for IEEE conferences) I was often asked to review 20+
| papers in a short time to determine who gets a full paper, who
| gets to present just a poster, etc. There was normally a second
| round, but often these would just look at submissions near the
| cutoff margin in the rankings. Obvious slop can be quickly
| rejected, but it will be easier to sneak things in.
| cyber_kinetist wrote:
| AI conferences are _already_ fucked. Students who are doing
| their Master 's degrees are reviewing those top-tier papers,
| since there are just too many submissions for existing
| reviewers.
| currymj wrote:
| Especially for your first NeurIPS paper as a PhD student, getting
| one published is extremely lucrative.
|
| Most big tech PhD intern job postings have NeurIPS/ICML/ICLR/etc.
| first author paper as a de facto requirement to be considered.
| It's like getting your SAG card.
|
| If you get one of these internships, it effectively doubles or
| triples your salary that year right away. You will make more in
| that summer than your PhD stipend. Plus you can now apply in
| future summers and the jobs will be easier to get. And it sets
| your career on a good path.
|
| A conservative estimate of the discounted cash value of a
| student's first NeurIPS paper would certainly be five figures.
| It's potentially much higher depending on how you think about it,
| considering potential path dependent impacts on future career
| opportunities.
|
| We should not be surprised to see cheating. Nonetheless, it's
| really bad for science that these attempts get through. I also
| expect some people did make legitimate mistakes letting AI touch
| their .bib.
| Der_Einzige wrote:
| This is 100% true, if anything you're massively undercounting
| the value of publications.
|
| Most industry AI jobs that aren't research based know that
| NeurIPS publications are a huge deal. Many of the managers
| don't even know what a workshop is (so you can pass off NeurIPS
| workshop work as just "NeurIPS")
|
| A single first author main conference work effectively allows a
| non Ph.D holder to be treated like they have a Ph.d (be
| qualified for professional researcher jobs). This means that a
| decent engineer with 1 NeurIPS publication is easily worth
| 300K+ YOY assuming US citizen. Even if all they have is a BS ;)
|
| And if you are lucky to get a spotlight or an oral, that's
| probably worth closer to 7 figures...
| alcasa wrote:
| Didn't know the L in Samuel L Jackson was for LeCun.
| j2kun wrote:
| I spot-checked one of the flagged papers (from Google, co-
| authored by a colleague of mine)
|
| The paper was https://openreview.net/forum?id=0ZnXGzLcOg and the
| problem flagged was "Two authors are omitted and one (Kyle
| Richardson) is added. This paper was published at ICLR 2024."
| I.e., for one cited paper, the author list was off and the venue
| was wrong. And this citation was mentioned in the background
| section of the paper, and not fundamental to the validity of the
| paper. So the citation was not fabricated, but it was incorrectly
| attributed (perhaps via use of an AI autocomplete).
|
| I think there are some egregious papers in their dataset, and
| this error does make me pause to wonder how much of the rest of
| the paper used AI assistance. That said, the "single error"
| papers in the dataset seem similar to the one I checked:
| relatively harmless and minor errors (which would be immediately
| caught by a DOI checker), and so I have to assume some of these
| were included in the dataset mainly to amplify the author's
| product pitch. It succeeded.
| davidguetta wrote:
| Yeah even the entire "Jane Doe / Jame Smith" my first thought
| is that it could have been a latex default value
|
| There was dumb stuff like this before the GPT era, it's far
| from convincing
| nativeit wrote:
| > Between 2020 and 2025, submissions to NeurIPS increased
| more than 220% from 9,467 to 21,575. In response, organizers
| have had to recruit ever greater numbers of reviewers,
| resulting in issues of oversight, expertise alignment,
| negligence, and even fraud.
|
| I don't think the point being made is "errors didn't happen
| pre-GPT", rather the tasks of detecting errors have become
| increasingly difficult because of the associated effects of
| GPT.
| ctoth wrote:
| > rather the tasks of detecting errors have become
| increasingly difficult because of the associated effects of
| GPT.
|
| Did the increase to submissions to NeurIPS from 2020 to
| 2025 happen because ChatGPT came out in November of 2022?
| Or was AI getting hotter and hotter during this period,
| thereby naturally increasing submissions to ... an AI
| conference?
| amitav1 wrote:
| I guess the way one would verify that this is more
| general trend in academia would be to run this on
| accepted papers to a non-AI conference?
| ls612 wrote:
| There are people who just want to punish academics for the
| sake of punishing academics. Look at all the people
| downthread salivating over blacklisting or even criminally
| charging people who make errors like this with felony fraud.
| Its the perfect brew of anti AI and anti academia sentiment.
|
| Also, in my field (economics), by far the biggest source of
| finding old papers invalid (or less valid, most papers state
| multiple results) is good old fashioned coding bugs. I'd like
| to see the software engineers on this site say with a
| straight face that writing bugs should lead to jail time.
| worik wrote:
| > I'd like to see the software engineers on this site say
| with a straight face that writing bugs should lead to jail
| time.
|
| My hand is up.
|
| I do not believe in gaol, but I do agree with the
| sentiment.
| ls612 wrote:
| Let he who is without sin cast the first stone...
| girvo wrote:
| If there were real consequences, we wouldn't be forced to
| churn out buggy nonsense by our employers. So we'd be
| able to take the time to do the right thing. Bug free
| software is possible, the world just says its not worth
| it today.
| ls612 wrote:
| >Bug free software is possible, ...
|
| Mr. Turing and his halting problem would like to politely
| disagree with this assertion.
| miki123211 wrote:
| And research codebases (in AI and otherwise) are usually of
| extremely bad quality. It's usually a bunch of extremely
| poorly-written scripts, with no indication which order to
| run them in, how inputs and outputs should flow between
| them, and which specific files the scripts were run on to
| calculate the statistics presented in the paper.
| nativeit wrote:
| I see your point, but I don't see where the author makes any
| claims about the specifics of the hallucinations, or their
| impact on the papers' broader validity. Indeed, I would have
| found the removal of supposed "innocuous" examples to be _far_
| more deceptive than simply calling a spade a spade, and
| allowing the data to speak for itself.
| gowld wrote:
| The point is that they should focus on the meaningful errors,
| not the automiation of meaningless errors.
| reliabilityguy wrote:
| Why these are meaningless? How do I know now that the whole
| paper is not a slop?
| j2kun wrote:
| The author calls the mistakes "confirmed hallucinations"
| without proof (just more or less evidence). The data never
| "speak for itself." The author curates the data and crafts a
| story about it. This story presented here is very suggestive
| (even using the term "hallucination" is suggestive). But
| calling it "100 suspected hallucinations", or "25 very likely
| hallucinations" does less for the author's end goal: selling
| their service.
| m-schuetz wrote:
| Bibtex are often also incorrectly generated. E.g., google
| scholar sometimes puts the names of the editors instead of the
| authors into the bibtex entry.
| worik wrote:
| > Bibtex are often also incorrectly generated
|
| ...and including the erroneous entry is squarely the author's
| fault.
|
| Papers should be carefully crafted, not churned out.
|
| I guess that makes me sweetly naive
| daveFNbuck wrote:
| You want the content of the paper to be carefully crafted.
| Bibtex entries are the sort of thing you want people to
| copy and paste from a trusted source, as they can be
| difficult to do consistently correctly.
| tuckerman wrote:
| I don't think the original comment was saying this isn't a
| problem but that flagging it as a hallucination from an LLM
| is a much more serious allegation. In this case, it also
| seems like it was done to market a paid product which makes
| the collateral damage less tolerable in my opinion.
|
| > Papers should be carefully crafted, not churned out.
|
| I think you can say the same thing for code and yet, even
| with code review, bugs slip by. People aren't perfect and
| problems happen. Trying to prevent 100% of problems is
| usually a bad cost/benefit trade-off.
| m-schuetz wrote:
| That's not happening for a similar reason people do not
| bug-check every single line of every single third-party
| library in their code. It's a chore that costs valuable
| time that you can instead spend on getting the actual stuff
| done. What's really important is that the scientific
| contribution is 100% correct and solid. For the references,
| the "good enough" paradigm applies. They mustn't be
| complete bogus, like the referenced work not existing at
| all which would indicate that the authors didnt even look
| at the reference. But minor issues like typos or rare
| issues with wrong authors can happen.
| bonzini wrote:
| To be honest, validating bibliographies does not cost
| valuable time. Every research group will have their own
| bibtex file to which every paper the group ever cited is
| added.
|
| Typically when you add it you get the info from another
| paper or copy the bibtex entry from Google scholar, but
| it's really at most 10 minutes work, more likely 2-5.
| Every paper might have 5-10 new entries in the
| bibliography, so that's 1 hour or less of work?
| miki123211 wrote:
| What's the benefit to society of making sure that academics
| waste even more of their valuable hours verifying that
| Google Scholar did not include extraneous authors in some
| citation which is barely even relevant to their work? With
| search engines being as good as they are, it's not like we
| can't easily find that paper anyway.
|
| The entire idea of super-detailed citations is itself quite
| outdated in my view. Sure, citing the work you rely on is
| important, but that could be done just as well via
| hyperlinks. It's not like anybody (exclusively) relies on
| printed versions any more.
| i_am_proteus wrote:
| >this error does make me pause to wonder how much of the rest
| of the paper used AI assistance
|
| And this is what's operative here. The error spotted, the
| entire _class_ of error spotted, is easily checked /verified by
| a non-domain expert. These are the errors we can confirm
| readily, with obvious and unmistakable signature of
| hallucination.
|
| If these are the only errors, we are not troubled. However: we
| do not know if these are the only errors, they are merely a
| signature that the paper was submitted without being thoroughly
| checked for hallucinations. They are a signature that some LLM
| was used to generate parts of the paper and the responsible
| authors used this LLM without care.
|
| Checking the rest of the paper requires domain expertise,
| perhaps requires an attempt at reproducing the authors'
| results. That the rest of the paper is now in doubt, and that
| this problem is so widespread, threatens the validity of the
| fundamental activity these papers represent: research.
| ls612 wrote:
| Google scholar and the vagaries of copy/paste errors has
| mangled bibitex ever since it became a thing, a single
| citation with these sorts of errors may not even be AI, just
| "normal" mistakes.
| fn-mote wrote:
| This seems like finding spelling errors and using them to
| cast the entire paper into doubt.
|
| I am unconvinced that the particular error mentioned above is
| a hallucination, and even less convinced that it is a sign of
| some kind of rampant use of AI.
|
| I hope to find better examples later in the comment section.
| j2kun wrote:
| I actually believe it was an AI hallucination, but I agree
| with you that it seems the problem is far more concentrated
| to a few select papers (e.g., one paper made up more than
| 10% of the detected errors).
| gold23 wrote:
| Why don't you look at the actual article? There are several
| more egregious examples, e.g., the authors being cited as
| "John Smith and Jane Doe"
| recursive wrote:
| What's the big deal with one dead canary? This coal mine's
| productivity is at record highs!
| jvanderbot wrote:
| The problem is, 10 years ago when I was still publishing even
| I would let an incorrect citation go through b/c of an old
| bibtex file or some such.
| neilv wrote:
| > _If these are the only errors, we are not troubled.
| However: we do not know if these are the only errors, they
| are merely a signature that the paper was submitted without
| being thoroughly checked for hallucinations. They are a
| signature that some LLM was used to generate parts of the
| paper and the responsible authors used this LLM without
| care._
|
| I am troubled by people using an LLM at all to write academic
| research papers.
|
| It's a shoddy, irresponsible way to work. And also
| plagiarism, when you claim authorship of it.
|
| I'd see a failure of the 'author' to catch hallucinations, to
| be more like a failure to hide evidence of misconduct.
|
| If academic venues are saying that using an LLM to write your
| papers is OK ("so long as you look it over for
| hallucinations"?), then those academic venues deserve every
| bit of operational pain and damaged reputation that will
| result.
| nearbuy wrote:
| The rate here (about 1% of papers) just doesn't seem that bad,
| especially if many of the errors are minor and don't affect the
| validity of the results. In other fields, over half of high-
| impact studies don't replicate.
| currymj wrote:
| the earlier list of ICLR papers had way more egregious
| examples. Those were taken from the list of submissions not
| accepted papers however.
| nazgul17 wrote:
| The thing is, when you copy paste a bibliography entry from the
| publisher or from Google Scholar, the authors won't be wrong.
| In this case, it is. If I were to write a paper with AI, I
| would at least manage the bibliography by hand, conscious of
| hallucinations. The fact that the hallucination is in the
| bibliography is a pretty strong indicator that the paper was
| written entirely with AI.
| arjvik wrote:
| I'm not sure I agree... while I don't ever see myself writing
| papers with AI, I hate wrangling a bibtex bibliography.
|
| I wouldn't trust today's GPT-5-with-web-search to do turn a
| bullet point list of papers into proper citations without
| checking myself, but maybe I will trust GPT-X-plus-agent to
| do this.
| jmmcd wrote:
| Google Scholar provides imperfect citations - very often
| wrong article type (eg article versus conference paper), but
| up to and including missing authors, in my experience.
| David_Osipov wrote:
| Great job! I've tried to test their tool as well, but was
| totally paywalled.
| fmbb wrote:
| > So the citation was not fabricated, but it was incorrectly
| attributed (perhaps via use of an AI autocomplete).
|
| Well the title says "hallucinations", not "fabrications". What
| you describe sounds exactly like what AI builders call
| hallucinations.
| _alternator_ wrote:
| The missing analysis is, of course, a comparison with pre-LLM
| conferences, like 2022 or 2023 that would show a "false
| positive" rate for the tool.
| cyber_kinetist wrote:
| It has been several years since the reviewing process for top AI
| conferences have been broken as hell, due to having too many
| submissions and only a few reviewers (up to the point that
| Masters students are reviewing the papers). It was only a matter
| of time before these conferences will be filled with AI-written
| papers.
| mat_b wrote:
| > we discovered 100s of hallucinated citations missed by the 3+
| reviewers who evaluated each paper.
|
| This says just as much about the humans involved.
| mkehrt wrote:
| Well for one, it's definitely not the responsibility of the
| reviewers to check that all the citations exist. That would be
| insane.
| Prof_Sigmund wrote:
| The authors talk about "a model's ability to align with human
| decisions" as a matter of the past. The omission in the paper is
| RLHF (Reinforcement Learning from Human Feedback). All these
| companies are "teaching machines to predict the preferences of
| people who click 'Accept All Cookies' without reading," by using
| low-paid human evaluators -- "AI teachers."
|
| If we go back to Google, before its transformation into an AI
| powerhouse -- as it gutted its own SERPs, shoving traditional
| blue links below AI-generated overlords that synthesize answers
| from the web's underbelly, often leaving publishers starving for
| clicks in a zero-click apocalypse -- what was happening?
|
| The same kind of human "evaluators" were ranking pages. Pushing
| garbage forward. The same thing is happening with AI. As much as
| the human "evaluators" trained search engines to elevate
| clickbait, the very same humans now train large language models
| to mimic the judgment of those very same evaluators. A feedback
| loop of mediocrity -- supervised by the... well, not the best
| among us. The machines still, as Stephen Wolfram wrote, for any
| given sequence, use the same probability method (e.g., "The cat
| sat on the..."), in which the model doesn't just pick one word.
| It calculates a probability score for every single word in its
| vast vocabulary (e.g., "mat" = 40% chance, "floor" = 15%, "car" =
| 0.01%), and voila! -- you have a "creative" text: one of a
| gazillion mindlessly produced, soulless, garbage "vile bile"
| sludge emissions that pollute our collective brains and render us
| a bunch of idiots, ready to swallow any corporate poison sent our
| way.
|
| In my opinion, even worse: the corporates are pushing toward
| "safety" (likely from lawsuits), and the AI systems are trained
| to sell, soothe, and please -- not to think, or enhance our
| collective experience.
| Lerc wrote:
| So the headline says
|
| > _GPTZero finds 100 new hallucinations in NeurIPS 2025 accepted
| papers_
|
| And I'm left wondering if they mean 100 papers or 100
| hallucinations
|
| The subheading says
|
| > _GPTZero 's analysis 4841 papers accepted by NeurIPS 2025 show
| there are at least 100 with confirmed hallucinations _
|
| Which accidentally a word, but seems to clarify that they do
| legitimately mean 100 papers.
|
| A later heading says
|
| > _Table of 100 Hallucinated Citations in Published Across 53
| NeurIPS Papers_
|
| Which suggests either the opposite, or that they chose a subset
| of their findings to point out a coincidentally similar number of
| incidents.
|
| How many papers did they find hallucinations in? I'm still not
| certain. Is it 100, 53 or some other number altogether? Does
| their quality of scrutiny match the quality of their
| communication. If they did in-fact find 100 Hallucinations in 53
| papers, would the inconsistency against their claim of "papers
| accepted by NeurIPS 2025 show there are at least 100 with
| confirmed hallucinations" meet their own bar for a hallucination?
| OptionX wrote:
| The old create the problem and sell the solution shtick.
| gowld wrote:
| Why does "Robust Label Proportions Learning" have a "Scan" link,
| while all the others have a "Sources" link? Was this web page
| generated by AI?
| gowld wrote:
| "100 Hallucinated Citations in Published Across 53 NeurIPS
| Papers"
|
| No one cares about citations. They are hallucinated because they
| are required to be present for political reasons, even though
| they have no relevance.
| gowld wrote:
| I searched Google for one of the hallucinations: [N. Flammarion.
| Chen "sam generalizes"]
|
| AI Overview: Based on the research, [Chen and N. Flammarion
| (2022)](https://gptzero.me/news/neurips/) investigate why
| Sharpness-Aware Minimization (SAM) generalizes better than SGD,
| focusing on optimization perspectives
|
| The link is a link to the OP web page calling the "research" a
| hallucination.
| djoldman wrote:
| I would love to see this analysis run on pre-GPT era papers.
| waldarbeiter wrote:
| My website of choice whenever I have to deal with references is
| dblp [1]. In my opinion more reliable than Google scholar in
| creating correct BibTeX. Also when searching for a paper you
| clearly see where it has been published or if it is only on
| arxiv.
|
| [1] https://dblp.org/
| sdellis wrote:
| This is an advertisement disguised as a "report".
| gold23 wrote:
| Nonetheless this investigation is important to anyone with a
| stronger desire to preserve intellectual honesty than disdain
| for a company trying to expand their offering.
| SaaSasaurus wrote:
| I'm surprised it's only 100, honestly. Also feels a little
| sensationalized... Before AI I wonder how many "hallucinations"
| were in human-written papers. Is there any data on this?
| rovr138 wrote:
| These are 100, already reviewed papers and accepted papers, by
| researchers in their areas of expertise. Usually PhD's and
| Professors.... They judge.
|
| These are not all the submissions that they received. The
| review process can be... brutal for some people (depending on
| the quality of their submission)
| abalone wrote:
| At least in one case the authors claimed to use ChatGPT to
| "generate the citations after giving it author-year in-text
| citations, titles, or their paraphrases." They pasted the
| hallucinations in without checking. They've since responded with
| corrections to real papers that in most cases are very similar to
| the hallucination, lending credibility to their claim.[1]
|
| Not great, but to be clear this is different from fabricating the
| whole paper or the authors inventing the citations. (In this case
| at least.)
|
| [1] https://openreview.net/forum?id=IiEtQPGVyV
| lifetimerubyist wrote:
| Surely this will help with the trust in our institutions that has
| been completely eroded over the last 5 years.
| thestructuralme wrote:
| The most striking part of the report isn't just the 100
| hallucinations--it's the "submission tsunami" (220% increase
| since 2020) that made this possible. We're seeing a literal
| manifestation of a system being exhausted by simulation.
|
| When a reviewer is outgunned by the volume of generative slop,
| the structure of peer review collapses because it was designed
| for human-to-human accountability, not for verifying high-speed
| statistical mimicry. In these papers, the hallucinations are a
| dead giveaway of a total decoupling of intelligence from any
| underlying "self" or presence. The machine calculates a
| plausible-looking citation, and an exhausted reviewer fails to
| notice the "Soul" of the research is missing.
|
| It feels like we're entering a loop where the simulation is
| validated by the system, which then becomes the training data for
| the next generation of simulation. At that point, the human
| element of research isn't just obscured--it's rendered
| computationally irrelevant.
___________________________________________________________________
(page generated 2026-01-22 23:00 UTC)