[HN Gopher] Entities enabling scientific fraud at scale (2025)
___________________________________________________________________
Entities enabling scientific fraud at scale (2025)
Author : peyton
Score : 244 points
Date : 2026-03-11 13:32 UTC (9 hours ago)
(HTM) web link (doi.org)
(TXT) w3m dump (doi.org)
| pixl97 wrote:
| This is Goodhart's law at scale. Number of released papers/number
| of citations is a target. Correctness of those papers/citations
| is much more difficult so is not being used as a measure.
|
| With that said, due to the apparent sizes of the fraud networks
| I'm not sure this will be easy to address. Having some kind of
| kill flag for individuals found to have committed fraud will be
| needed, but with nation state backing and the size of the groups
| this may quickly turn into a tit for tat where fraud accusations
| may not end up being an accurate signal.
|
| May you live in interesting times.
| armchairhacker wrote:
| There's an accurate way to confirm fraud: look for
| inconsistencies and replicate experiments.
|
| If the fraudsters "fail to replicate" legitimate experiments,
| ask them for details/proof, and replicate the experiment
| yourself while providing more details/proof. Either they're
| running a different experiment, their details have
| inconsistencies, or they have unreasonable omissions.
| wswope wrote:
| Yeah, but this happens all the time.
|
| >>95% of the time, the fraudsters get off scot-free. Look at
| Dan Ariely: Caught red-handed faking data in Excel using the
| stupidest approach imaginable, and outed as a sex pest in the
| Epstein files. Duke is still giving him their full backing.
|
| It's easy to find fraud, but what's the point if our
| institutions have rotten all the way through and don't care,
| even when there's a smoking gun?
| pixl97 wrote:
| Of course this is slightly messy too. Fraudsters are probably
| always incorrect, of course they could have stolen the data.
| But being incorrect doesn't mean your intentionally
| committing fraud.
| john_strinlai wrote:
| that approach is accurate, but not scalable.
|
| the effort to publish a fraudulent study is less (sometimes
| much less) than the effort to replicate a study.
| awesome_dude wrote:
| Is it that easy?
|
| Machine Learning papers, for example, used to have a terrible
| reputation for being inconsistent and impossible to
| replicate.
|
| That didn't make them (all) fraudulent, because that requires
| intent to deceive.
| itintheory wrote:
| What do you think it is about machine learning that makes
| it hard to replicate? I'm an outsider to academic research,
| but it seems like computer based science would be uniquely
| easy - publish the code, publish the data, and let other
| people run it. Unless it's a matter of scale, or access to
| specific hardware.
| renewiltord wrote:
| A lot of things are easy if you ignore the incentive
| structure. E.g. a lot of papers will no longer be
| published if the data must be published. You'd lose all
| published research from ML labs. Many people like you
| would say "that's perfectly okay; we don't need them" but
| others prefer to be able to see papers like Language
| Models Are Few-Shot Learners
| https://arxiv.org/abs/2005.14165
|
| So the answer is that we still want to see a lot of the
| papers we currently see because knowing the technique
| helps a lot. So it's fine to lose replicability here for
| us. I'd rather have that paper than replicability through
| dataset openness.
| armchairhacker wrote:
| But the lab must publish at least the general category of
| data, and if that doesn't replicate, then the model only
| works on a more specific category than they claim (e.g.
| only their dataset).
| awesome_dude wrote:
| Even with the exact same dataset and architecture, ML
| results aren't perfectly replicable due to random weight
| initialisation, training data order, and non-
| deterministic GPU operations. I've trained identical
| networks on identical data and gotten different final
| weights and performance metrics.
|
| This doesn't mean the model only works on that specific
| dataset - it means ML training is inherently stochastic.
| The question isn't 'can you get identical results' but
| 'can you get comparable performance on similar data
| distributions.
| armchairhacker wrote:
| Then researchers should re-train their models a couple
| times, and if they can't get consistent results, figure
| out why. This doesn't even mean they must throw out the
| work: a paper "here's why our replications failed"
| followed by "here's how to eliminate the failure" or
| "here's why our study is wrong" is useful for future
| experiments and deserves publication.
| awesome_dude wrote:
| As per my previous comment - we are discussing stochastic
| systems.
|
| By definition, they involve variance that cannot be
| explained or eliminated through simple repetition.
| Demanding a 'deterministic' explanation for stochastic
| noise is a category error; it's like asking a
| meteorologist to explain why a specific raindrop fell an
| inch to the left during a storm replication.
| ertgbnm wrote:
| That would be great if journals bothered publishing
| replication studies. But since they don't, researchers can't
| get adequate funding to perform them, and since they can't
| perform them, they don't exist.
|
| We can't look for failed replication experiments if none
| exist.
| bwfan123 wrote:
| > This is Goodhart's law at scale.
|
| Also, Brandolini's law. And Adam Smith's law of supply and
| demand. When the ability to produce overwhelms the ability to
| review or refute, it cheapens the product.
| otherme123 wrote:
| > Number of released papers/number of citations is a target
|
| There was this guy, well connected in the science world, that
| managed to publish a poor study quite high (PNAS level). It was
| not fraud, just bad science. There were dozens of papers and
| letters refuting his claims, highlighting mistakes, and so...
| Guess what? Attending to metrics (citations, don't matter if
| they are citing you to say you were wrong and should retract
| the paper!), the original paper was even more stellar on the
| eyes of grants and the journal itself.
|
| It was rage bait before Facebook even existed.
| bonoboTP wrote:
| > Number of released papers/number of citations is a target
|
| Only in stupid university leaderships is that truly what gets
| you hired or promoted. It's simply not true. Junior researchers
| in fact are believing it stronger than the facts actually
| support. Yes, you have to have a solid amount of publications,
| but doing a ridiculous amount of low-impact salami-sliced stuff
| or getting your name on a ton of papers where you did no real
| work is not going to win you a job. People who evaluate
| applications also live in this world and know that these
| metrics are being gamed. It's a cat and mouse game but the cats
| are also paying attention. You can only play this against
| really dumb government bureaucracies that mechanically give
| points for publications and have hard numerical criteria etc.
| Good institutions don't do that.
|
| Good evaluators actually read the papers themselves. Of course
| you can't read the papers of every single applicant if there
| are many. But once the applicant gets into the a somewhat
| filtered down list, reading the paper(s) or having an interview
| about it, or having them give a talk is much more informative
| than the number of the papers. Still not perfect, because some
| people can't communicate well, but communicating is part of the
| job, so maybe that's super bad but somewhat bad.
|
| Evaluators will use also other evidence such as recommendation
| letters (informally being aware of the reputation of the
| recommender), previous fellowships or grants obtained, etc.
|
| None of these are foolproof in themselves. But someone who has
| super few publications relative to their career stage will need
| some other piece of evidence in favor.
|
| In machine learning and AI, peer reviews are known to be quite
| random. If you have a good Arxiv-only paper that makes sense
| and you can give a good talk on it and answer questions, that
| will get you further than having a rubberstamp on some paper
| that's "meh, so what".
|
| There are some players in this game (which includes funding
| agencies, journals, university administration, hiring
| committees, conference organizers, students, etc) that are more
| ossified and slow-moving than others.
|
| And it's also true that double blind peer review and the
| rubberstamp of a top-tier conference was mostly beneficial to
| small, not well connected research groups, as it puts the paper
| on an equal footing with the big labs. The more this system
| erodes, to more we fall back to reputation and branding of big
| labs and famous researchers. Again, because there is no
| infinite time and infinite wisdom available to pick from
| applicants and there never will be. There are only tradeoffs.
| gjsman-1000 wrote:
| The future of science, the Internet, and all things: The Library
| of Babel by Jorge Luis Borges.
|
| Some things should not have been democratized. Silicon Valley
| assumes that removing restrictions on information brings freedom,
| but reality shows that was naive.
| honeycrispy wrote:
| You shouldn't just assume that the inverse would be free from
| fraud. The incentives for fraud still apply even when the
| system is not democratized.
| gjsman-1000 wrote:
| Except with AI, a fraudulent gatekept world would still be a
| smaller percentage of fraud than what is coming. Infinite
| scale fraud.
|
| The soviets may have rigged a few studies; but the
| democratized world now faces almost all studies being rigged.
| honeycrispy wrote:
| I think it'd be a different form of fraud that would be
| much harder to discredit. Think sugar industry blaming fat
| for health issues. More of that.
| rdevilla wrote:
| Tearing down gatekeeping (i.e. "high standards") in pursuit of
| maximal inclusivity is just another way of saying "regression
| to the mean."
|
| The gate has been removed from the signal chain, and now the
| noise floor is at infinity.
| qsera wrote:
| There is a saying in my native language that goes something
| like "If you mix poison and milk, the milk will turn
| poisonous, instead of poison becoming milk (aka beneficial)".
|
| I guess, to convert it into this context, we can say that if
| you mix the high minded and infantile (which I think is what
| Internet and social media did), the high minded becomes
| infantile, instead of the other way around.
| convolvatron wrote:
| there is no 'sin of maximal inclusivity here', the gate is
| broken, but primarily because it was largely an honor system
| before, and no one has the motivation or resources to really
| dig into a lot of these papers.
|
| in no sense was it corrupted by the desire to include a
| larger population in journal publications.
| leoc wrote:
| In what way was it was democratised? We're not talking about
| Substacks and YouTube channels here, we're not even talking
| about arXiv preprints and the like, we're talking about peer-
| reviewed journal publications, and that system remains gated in
| much the same way that it was in the 1980s when it comes to
| trying to publish in it. If anything this system is the poster
| child for top-down gatekeeping by the recognised authorities,
| and it's precisely the value of that official recognition that
| makes people so desperate to break into it. The major changes
| seem to have been the easy availability of author publication
| lists and the advent of publication metrics, not things which
| have been or were ever meant to be particularly democratising
| for would-be authors; and an increase in the number of people
| playing the game, driven to a large extent by increasing
| participation from developing countries, and hopefully not many
| people would have the gall to argue for a ban on developing-
| country participation.
| niam wrote:
| The Library of Babel comparison is too fatalistic imo, even
| granting that it's maybe just an extreme example. The real
| world doesn't quite resemble a closed system with no metadata.
| We can still establish chains of trust.
|
| Whether or not people will build resilient chains is another
| story, contingent on whether the strength of that chain
| actually matters to people. It probably doesn't for a lot of
| people. Boo. But inasmuch as _I_ care, I feel I ought to be
| free to try and derive a strong signal through the noise.
| RobotToaster wrote:
| It kinda skips over how large mainstream journals, with their
| restrictive and often arbitrary standards, have contributed to
| this. Most will refuse to publish replications, negative studies,
| or anything they deem unimportant, even if the study was
| conducted correctly.
| tppiotrowski wrote:
| Maybe we need a journal completely dedicated to replication
| studies? It would attract a lot of attention I think.
| pfdietz wrote:
| And _funding_ dedicated to replication studies.
| fsckboy wrote:
| paid by the original authors if their study fails to
| replicate
| MichaelDickens wrote:
| Economics has the Journal of Comments and Replications in
| Economics: https://jcr-econ.org/
| LargeWu wrote:
| Is there a viable career path for researchers who choose to
| focus on replication instead of novel discoveries? I assume
| replications are perceived as less prestigious, but it's also
| important work.
| stanford_labrat wrote:
| sadly no, this is not a thing and it's critically needed.
|
| top on my list of things to do if i were a billionaire:
| launch an institute for the sole purpose of reproducing
| other's findings.
| cvwright wrote:
| The closest thing we have is, in security / privacy /
| cryptography, you can write "attack" papers.
|
| It's not perfect. You don't get any credit unless you can
| demonstrate a substantial break of the prior work. But it's
| better than in a lot of other fields.
| fc417fc802 wrote:
| We already have archival journals. What's missing is funding
| and any prospect of career advancement.
| CGMthrowaway wrote:
| So much of this started with the rise of the peer-review
| journal cartel, beginning with Pergamon Press in 1951
| (coincidentally founded by Ghislaine Maxwell's father). "Peer
| review" didn't exist before then, science papers and discussion
| was published openly, and scientists focused on quality not
| quantity.
| john_strinlai wrote:
| > _Pergamon Press in 1951 (coincidentally founded by
| Ghislaine Maxwell 's father)_
|
| perhaps a bit off-topic, but what is coincidental about this
| and/or what is the relevance of Ghislaine Maxwell here?
| benterix wrote:
| It's useless, but I'm ashamed to admit I found this tiny
| piece of trivia interesting.
| readthenotes1 wrote:
| Like the paywall blocking many scientific arti6, perhaps
| it would be best if we released also the Epstein Files?
| anonymars wrote:
| I imagine it's the interesting peculiarity that the same
| people seem to crop up over and over and over again. Six
| degrees of Kevin Bacon or something, except it's like one
| or two degrees. As George Carlin said, "it's a big club,
| and you ain't in it"
|
| For example Donald Barr (father of twice-former US Attorney
| General Bill Barr) hiring college-dropout Jeffrey Epstein
| whilst headmaster at the elite Dalton School
|
| Additional fun facts about Donald Barr: he served in US
| intelligence during WWII, and wrote a sci-fi book featuring
| child sex slaves
| jl6 wrote:
| Also the Epstein-Barr virus causes Mono, the clone of
| .NET, which was created by Bill Gates, known associate of
| Epstein, whose father was president of the Washington
| State Bar Association. And you know who _else_ works in
| Washington? Join the dots, people.
| Henchman21 wrote:
| This might be my fav HN comment ever. Well done!
| underlipton wrote:
| We call people who make connections like these
| "conspiracy theorists," until they're right, at which
| point we call them "right". And somewhere in between, if
| they manage to get a job, we call them "Simpsons
| writers."
| tialaramex wrote:
| Ghislaine's father (Robert Maxwell) was _also_ a terrible
| person but for different reasons.
|
| Robert Maxwell was a crook, he used pension funds (supposed
| to be ring-fenced for the benefit of the pensioners) to
| prop up his companies, so, after his slightly mysterious
| death it was discovered that basically there's no money to
| pay people who've been assured of a pension when they
| retire.
|
| He was also very litigious. If you said he was a crook
| _when he was alive_ you 'd better hope you can prove it
| _and_ that you have funding to stay in the fight until you
| do. So this means the sort of people who call out crooks
| were _especially_ unhappy about Robert Maxwell because he
| was a crook _and_ he might sue you if you pointed it out.
| bryanrasmussen wrote:
| I believe by saying it is coincidental they are saying
| there is probably no relevance, just an interesting piece
| of trivia, why put out this interesting piece of trivia?
| Because maybe someone will be able to make an argument of
| relevance.
| pocksuppet wrote:
| It's more than coincidental, but tangential to the point.
| It shows crime runs in families.
| bartread wrote:
| If you want to know more about the history of Pergamon
| Press there's a great Behind the Bastards episode on Robert
| Maxwell (Ghislaine Maxwell's father) - who himself was a
| scumbag in a variety of ways that were entirely distinct
| from Ghislaine Maxwell's brand of scumbaggery - that covers
| this. Might even be a multipart episode - it's a while
| since I've listened to it, but I have a feeling it's at
| least a two parter.
| pessimizer wrote:
| "Coincidental" means random, with no causal connection
| being explicitly claimed. It just means that two things
| share some characteristic (such as being relatives.) The
| thing that is coincidental is that the person who founded
| the company being discussed is also the father of another
| person who current events have brought into prominence.
|
| It's why you would say something like "more than
| coincidental" if you were trying to make some causal claim,
| like one thing causing the other, or both things coming
| from the same cause.
|
| So, "What is coincidental about that?" is a weird question.
| It reads as a rhetorical claim of a causal connection
| through asking for a denial or a disproof of one.
| john_strinlai wrote:
| sorry.
|
| what is the relevance to the discussion about journals
| and peer review is my main question.
|
| if i randomly mentioned that your name appears to be an
| alternate spelling of a 3-band active EQ guitar pedal,
| coincidentally sharing all of the letters except one, in
| my reply to you, most people would be confused. that is
| how i felt when randomly reading "Ghislaine Maxwell" in
| this context of journals and peer review.
| leoc wrote:
| I'm not sure that the system was ever that near to
| perfection: for example, John Maddox of _Nature_ didn 't like
| the advent of pre-publication peer review, but that
| presumably had something to do with it limiting his
| discretion to approve and desk-reject whatever he wanted. But
| in any case it (like other aspects of the cozy interwar and
| then wartime scientific world) could surely never have
| survived the huge scaling-up that had already begun in the
| post-war era and created the pressure to switch to pre-
| publication peer reivew in the first place.
| throwaway27448 wrote:
| > coincidentally founded by Ghislaine Maxwell's father
|
| A crazy world we live in where Robert Maxwell's daughter is
| more notorious than he is.
| LarsDu88 wrote:
| Fun fact, he almost got the worldwide console rights to
| Tetris back in the 80s, and tried going to Soviet officials
| to get those rights. To the point he's the antagonist of a
| recent "Tetris" movie that came out.
| throwaway27448 wrote:
| This is a fun fact, thank you.
| lovich wrote:
| Never knew of the guy but what a terrible sounding person
| from his Wikipedia at least.
|
| Shit apple doesn't fall far from the shit tree I guess.
| butILoveLife wrote:
| >scientists focused on quality not quantity.
|
| I know a PhD professor doing post doc or something, and he
| accepted a scientific study just because it was published in
| Nature.
|
| He didn't look at methodology or data.
|
| From that point forward, I have never really respected
| Academia. They seem like bottom floor scientists who never
| truly understood the scientific method.
|
| It helped that a year later Ivys had their cheating scandals,
| fake data, and academia wide replication crisis.
| fc417fc802 wrote:
| When I read something in a textbook I blindly believe it,
| depending on the broader context and the textbook in
| question. Is that a bad thing?
|
| People are constantly filtering everything based on
| heuristics. The important thing is to know how deep to look
| in any given situation. Hopefully the person you're
| referring to is proficient at that.
|
| Keep in mind that research scientists need to keep abreast
| of far more developments than any human could possibly
| study in detail. Also that 50% of people are below average
| at their job.
| irishcoffee wrote:
| > When I read something in a textbook I blindly believe
| it, depending on the broader context and the textbook in
| question. Is that a bad thing?
|
| It is if what you read is factually incorrect, yes.
|
| For example, I have read in a textbook that the tongue
| has very specific regions for taste. This is patently
| false.
|
| > Keep in mind that research scientists need to keep
| abreast of far more developments than any human could
| possibly study in detail. Also that 50% of people are
| below average at their job.
|
| So, we should probably just discount half of what we read
| from research scientists as "bad at their job" and not
| pay much attention to it? Which half? Why are you
| defending corruption?
| cyberax wrote:
| The problem is that you can't just verify everything
| yourself. You likely have your own deadlines, and/or you
| want to do something more interesting than replicating
| statistical tests from a random paper.
| irishcoffee wrote:
| > The problem is that you can't just verify everything
| yourself.
|
| So the problem is reduced to "I believe what I want! This
| person said it and so I think it's true!"
|
| Sounds like politics in a nutshell.
| cyberax wrote:
| No, it's not. It's reduced to "I trust people from a
| respectable scientific journal with 150 years of
| history".
|
| > Sounds like politics in a nutshell.
|
| Again, no. It sounds like the division of labor. The
| thing that made modern human societies possible.
| irishcoffee wrote:
| Division of labor. Dividing labor between the "i'll pay
| you to work" and "I'm paid to work"
|
| The jokes write themselves,
| cyberax wrote:
| Yes? What is exactly funny here? This is literally how
| the civilization works. I'm paid to do my work, and I pay
| others to do their work.
|
| Do you grow your own food and sew your own clothes? Also,
| did you personally etch the microprocessor that runs your
| computer? The division of labor inherently means trusting
| others. So when I buy a bag of M4 screws, I'm not going
| to measure each screw with a micrometer, and I'm not
| taking X-ray spectra to verify their material
| composition.
|
| The academic world also used to trust large publishers to
| take care to actually review papers. It appears that this
| trust is now misplaced. But I don't think it was somehow
| stupid.
| fc417fc802 wrote:
| You don't seem to be engaging in good faith.
| Calavar wrote:
| This is a good point. It is not humanly possible to
| verify every claim you read from every source.
|
| Ideally, you should independently verify claims that
| appear to be particularly consequential or particularly
| questionable on the surface. But at some point you have
| to rely on heuristics like chain of trust (it was peer
| reviewed, it was published in a reputable textbook), or
| you will never make forward progress on anything.
| natpalmer1776 wrote:
| There is a vast difference between a student reading from
| a textbook and a researcher / scientist reading studies
| and/or papers.
|
| As a student you are to be directed* in your reading by
| an expert in the field of study that you are learning
| from. In many higher level courses a professor will
| assign multiple textbooks and assign reading from only
| particular chapters of those textbooks specifically
| because they have vetted those chapters for accuracy and
| alignment with their curriculum.
|
| As a researcher and scientist a very large portion of
| your job is _verifying and then integrating_ the research
| of others into your domain knowledge. The whole purpose
| of replicating studies is to look critically at the
| methodology of another scientist and try as hard as you
| can to prove them wrong. If you fail to prove them wrong
| and can produce the same results as them, they have done
| Good Science.
|
| A textbook is the product of scientists and researchers
| Doing Science and publishing their results, other
| scientists and researchers _verifying via replication_ ,
| and then one of those scientists or researchers who is an
| expert in the field doing their best to compile their
| knowledge on the domain into a factually accurate and
| (relatively) easy to understand summary of the collective
| research performed in a specific domain.
|
| The fact is that people make mistakes, and the job of a
| professor (who is an expert in a given field) is to
| identify what errors have made it through the various
| checks mentioned above and into circulation, often times
| making subjective judgement calls about what is 'factual
| enough' for the level of the class they are teaching, and
| leverage that to build a curriculum that is sound and
| helps elevate other individuals to the level of knowledge
| required to contribute to the ongoing scientific journey.
|
| _In short, it 's not a bad thing if you're learning a
| subject by yourself for your own purposes and are not
| contributing to scientific advancement or working as an
| educator in higher-education._
|
| * You can self-study, but to become an expert while doing
| so requires extremely keen discernment to be able to root
| out the common misconceptions that proliferate in any
| given field. In a blue-collar field this would be akin to
| picking up 'bad technique' by watching YouTube videos
| published by another self-taught tradesman; it's not
| always obvious when it happens.
| jruohonen wrote:
| > If you fail to prove them wrong and can produce the
| same results as them, they have done Good Science.
|
| Not really in my humble opinion. Sure, the Popperian vibe
| is kind of fundamental, but the whole truncation into
| binary-valued true/false categories seldom makes sense
| with many (or even most?) problems for which
| probabilities, effect sizes, and related things matter
| more.
|
| And if you fail to replicate a study, they may have still
| done Good Science. With replications, it should not be
| about Bad Science and Good Science but about the
| cumulation of evidence (or a lack thereof). That's what
| meta-analyses are about.
|
| When we talk about Bad Science, it is about the
| industrial-scale fraud the article is talking about. No
| one should waste time replicating, citing, or reading
| that.
| fc417fc802 wrote:
| > There is a vast difference between a student reading
| from a textbook and a researcher / scientist reading
| studies and/or papers.
|
| Not really. Both are learning new things. Neither has the
| time or access to resources to replicate even a small
| fraction of things learned. Neither will ever make direct
| use of the vast majority of things learned.
|
| Thus both depend on a cooperative model where trust is
| given to third parties to whom knowledge aggregation is
| outsourced. In that sense a textbook and prestigious peer
| reviewed journals serve the same purpose.
| bee_rider wrote:
| Academia has problems, like everywhere else. But that seems
| like a big extrapolation from just one professor.
|
| Fake data--you can only get that type of scandal when
| people are checking the data. I'd be more skeptical of
| communities that never have that kind of scandal.
| gus_massa wrote:
| Most of the times you don't "accept" results. You have to
| build something on them, like an extension or a similar
| version on other field. So usually the first step is try to
| understand the cryptic published version and do a
| reproduction or something as close as possible.
|
| The exact reproductions is never published, because
| journals don't accept them, but if you add a few tweaks
| here and there you have a nice seed for an article to
| publish somewhere.
|
| (I may "accept" an article in a field I don't care, but you
| probably should not thrust my opinion in fields I don't
| care.)
| underlipton wrote:
| Some "fun" reading on the subject of Mr. Maxwell:
|
| https://sarahkendzior.substack.com/p/red-lines
|
| tl;dr He is the bridge that uncomfortably links Biden's
| former Secretary of State, Antony Blinken, to Jeffrey Epstein
| and Mossad. Hence, * _gestures at the last couple of weeks
| and years_ *. Dude was just, like, Fraud Central, apparently.
| canjobear wrote:
| Peer review existed before 1951 in the US at least. See for
| example Einstein's reaction to negative reviews when he tried
| to publish in Physical Review in 1935
| https://paeditorial.co.uk/post/albert-einstein-what-did-
| he-t...
| jayde2767 wrote:
| I wish you had highlighted or bolded "cartel", which is
| exactly how those industry players act.
| leoc wrote:
| Right, it seems that many of the weaknesses in the system exist
| because they serve the interests of journal publishers or of
| normal, legitimate-ish researchers, but in the process open the
| door to full-time system-hackers and pure fraudsters.
| bonoboTP wrote:
| Any system that grows too fast has these kinds of problems.
| When it's a small intimate circle where everyone knows
| everyone, reputation alone can keep people in check. Once
| it's larger you need to invent rules and bureaucracies and
| structures and you will have loopholes that bad actors can
| more easily exploit, hiding in the crowd, than in the small
| version. It's the same with the Internet or computing.
| Security was much less of a topic when it was mostly honest
| academic nerds using the Internet, and the protocol designs
| often didn't even assume adversarial participants. Science
| also still runs on this assumed honesty system that worked
| well when it was small.
| ramraj07 wrote:
| Do you want issues of Nature and cell to be replication
| studies? As a reader even from within the field, im not
| interested in browsing through negative studies. It'll be great
| if I can look them up when needed but im not looking forward to
| email ToC alerts filled with them.
|
| Also who's funding you for replication work? Do you know the
| pressure you have in tenure track to have a consistent thesis
| on what you work on?
|
| Literally every single know that designs academia is tuned to
| not incentivize what you complain about. Its not just journals
| being picky.
|
| Also the people committing fraud aren't ones who will say "gosh
| I will replicate things now!" Replicating work is far more
| difficult than a lot of original work.
| zhdc1 wrote:
| > Do you want issues of Nature and cell to be replication
| studies? As a reader even from within the field, im not
| interested in browsing through negative studies.
|
| Actually, yes, I do. The marginal cost for publishing a study
| online at this point is essentially nil.
| recursivecaveat wrote:
| I think archives with pretty low standards for notability
| are a good idea. At some point though you have to pick what
| actually counts as interesting enough to go in the curated
| list that is actually suggested reading, where the prestige
| is attached. If there's no curation by Nature then it falls
| to bloggers or another journal to sift through the fire-
| hose and make best-of lists. Most of the value is in the
| curation, not the publishing. Without exclusivity there's
| very little signal.
| mmooss wrote:
| > The marginal cost for publishing a study online at this
| point is essentially nil.
|
| The marginal cost for doing a study remains the same, which
| is quite a bit. Society doesn't have unlimited scientific
| talent or hours. Every year someone spends replicating is a
| year lost to creating something new and valuable.
| Bratmon wrote:
| > Do you want issues of Nature and cell to be replication
| studies?
|
| More than anything. That might legitimately be enough to save
| science on its own.
| dekhn wrote:
| Maybe nature and cell and a few other journals should be
| exceptions: they should be the place that the most advanced
| scientists publish interesting ideas early for the
| consumption by their competitors. At that level of science,
| all the competitors can reproduce each other's experiments
| if necessary; the real value is expanding the knowledge of
| what seems possible quickly.
|
| (I am not seriously proposing this, but it's interesting to
| think about distinguishing between the very small amount of
| truly innovative discovery versus the very long tail of
| more routine methods development and filling out gaps in
| knowledge)
| Bratmon wrote:
| > that level of science, all the competitors can
| reproduce each other's experiments if necessary
|
| But they don't, and that's the problem!
| dekhn wrote:
| Advanced groups usually replicate their competitor's
| results in their own hands shortly after publication (or
| they just trust their competitor's competence). But they
| don't spend any time publishing it unless they fail to
| replicate and can explain why they can't replicate. From
| their perspective, it's a waste of time. I think this has
| been shown to be a naive approach (given the high rate of
| image fraud in molecular biology) but people who are in
| the top of the field have strong incentives to focus on
| moving the state of the art forward without expending
| energy on improving the field as a whole.
| Bratmon wrote:
| All that makes it more important for top journals to
| reward replication, not less!
| jltsiren wrote:
| Top journals are not inherently prestigious. They are
| prestigious because they try to publish only the most
| interesting and most significant results. If they started
| publishing successful replication studies, they would
| lose prestige, and more interesting journals would
| eventually rise to the top. (Replication studies that
| fail to replicate a major result in a spectacular way are
| another matter.)
| MarkusQ wrote:
| "strong incentives to focus on moving the state of the
| art forward without expending energy on improving the
| field as a whole"
|
| That sort of Orwellian doublethink is exactly the
| problem. They need to move it forward without improving
| it, contribute without adding anything, challenge
| accepted dogma without rocking the boat, and...blech!
| godelski wrote:
| > challenge accepted dogma without rocking the boat
|
| I think the funniest part is how we have all these heroes
| of science who faced scrutiny by their peers, but
| triumphed in the end. They struggled because they
| challenged the status quo. We celebrate their anti
| authoritative nature. We congratulate them for their
| pursuit of truth! And then get mad when it happens. We
| pretend this is a thing of the past, but it's as common
| as ever[0,1].
|
| You must create paradigm shifts without challenging the
| current paradigm!
|
| [0] https://www.scientificamerican.com/article/katalin-
| karikos-n...
|
| [1] https://www.globalperformanceinsights.com/post/how-a-
| rejecte...
| godelski wrote:
| Are you explaining this from experience or from
| speculation?
|
| I can tell you that it doesn't match my own experience. I
| also think it doesn't match your example. Those cases of
| verified image fraud are typically part of replication
| efforts. The reason the fraud is able to persist is due
| to the lack of replication, not the abundance of it.
| dekhn wrote:
| Mostly experience (based on being a PhD scientist, a
| postdoc, a National Lab scientist, and engineer at
| several bigtech companies), partly speculation (none of
| the groups/labs I worked in operated at "the highest
| level", but I worked adjacent to many of those).
|
| I'm pretty sure most image fraud went completely
| unrealized even in the case of replication failure. It
| looks like (pre AI) it was mostly a few folks who did it
| as a hobby, unrelated to their regular jobs/replication
| work.
| godelski wrote:
| In most of the labs I've worked in replication is not a
| common task[0] > 'm pretty sure most
| image fraud went completely unrealized even in the case
| of replication failure
|
| Part of my point is that being unable to publish
| replication efforts means we don't reduce ambiguity in
| the original experiments. I was taught that I should
| write a paper well enough that a PhD student (rather than
| candidate) should be able to reproduce the work. IME
| replication failures are often explained with "well I
| must be doing something wrong." A reasonable conclusion,
| but even if true the conclusion is that the original
| explanation was insufficiently clear. >
| It looks like (pre AI) it was mostly a few folks who did
| it as a hobby
|
| I'm sorry, didn't you say >>> Advanced
| groups usually replicate their competitor's results in
| their own hands shortly after publication
|
| Because your current statement seems to completely
| contradict your previous one.
|
| Or are you suggesting that the groups you didn't work
| with (and are thus speculating) are the ones who
| replicate works and the ones you did work with "just
| trust their competitor's competence")? Because if this is
| what you're saying then I do not think this "mostly"
| matches your experience. That your experience more
| closely matches my own.
|
| [0] I should take that back. I started in physics
| (undergrad) and went to CS for grad. Replication could
| often be de facto in physics, as it was a necessary step
| towards progress. You often couldn't improve an idea
| without understanding/replicating it (both theoretical
| and experimental). But my experience in CS, including at
| national labs, was that people didn't even run the code.
| Even when code was provided as part of reviewing
| artifacts I found that my fellow reviewers often didn't
| even look at it, let alone run it... This was common at
| tier 1 conferences mind you... I only knew one other
| person that consistently ran code.
| dekhn wrote:
| Note that my field is biophysics (quantitative biology)
| while yours is physics and CS. Those are done completely
| differently from biology; with the exception of some
| truly enormous/complex/delicate experiments that require
| unique hardware, physics tends to be much more
| reproducible than biology, and CS doubly-so.
|
| Replication of an experiment and finding image fraud are
| kind of done as two different things. If somebody
| publishes a paper with image fraud, it's still entirely
| possible to replicate their results(!) and if somebody
| publishes a paper without any image fraud, it's still
| entirely possible that others could fail to replicate.
| Also, most image errors in papers are, imho, due to
| sloppy handling/individual errors, rather than
| intentional fraud (it's one of the reasons I worked so
| hard on automating my papers- if I did make an error,
| there should be audit log demonstrating the problem, and
| the error should be rectified easily/quickly in the same
| way we fix bugs in production at big tech).
|
| This came up a bunch when I was at LBL because of work
| done by Mina Bissell there on extracellular matrix. She
| is actively rewriting the paradigm but many people can't
| reproduce her results- complex molecular biology is
| notororiously fickle. Usually the answer is, "if you're a
| good researcher and can't reproduce my work, you come to
| my lab and reproduce it there" because the variables that
| affect this are usually things in the lab- the
| temperature, the reagents, the handling.
|
| See https://www.nature.com/articles/503333a (written by
| Dr. Bissell).
| godelski wrote:
| > physics tends to be much more reproducible than
| biology, and CS doubly-so.
|
| With physics I think there is a better culture of
| reproduction, but that is, I believe, due more to
| culture. That it is acceptable to "be slow". There's a
| high stress on being methodical and extremely precise.
| The prestige is built on making your work bulletproof,
| and so you're really encouraged to help others reproduce
| your work as it strengthens it. You're also encouraged to
| analyze in detail and to faithfully reproduce, because
| finding cracks also yields prestige. I don't know if it's
| the money, but no one is in it for the money. Physics
| sure is a lot harder than anything else I've done and it
| pays like shit.
|
| For CS the problem is wildly different. It should be easy
| to reproduce as code is trivial to copy. Ignoring the
| issue of not publishing code alongside results, there's
| also often subtle things that can make or break works.
| I've found many times in replication efforts that the
| success can rely on a single line that essentially comes
| form a work that was the reference to a reference of the
| work I'm trying to reproduce. The problem here is
| honestly more of laziness. In contrast to physics there's
| an extreme need for speed. In physics (like everyone else
| I knew) I often felt like I was not smart enough, and
| that encouraged people to dive deeper and keep improving
| or to give up. In CS (like everyone else I knew) I often
| felt like I was not fast enough, and that encouraged
| people to chase sponsorships from labs that provided more
| compute, it encouraged a "shotgun" approach (try
| everything), or for people to give up (aka "GPU poor").
|
| The reason I'm saying this is because I think it is
| important to understand the different cultures and how
| replication efforts differ. In physics a replication
| failure was often assumed to be due to a lack of
| intelligence. In CS a replication effort is seen as a
| waste of time. Both are failures of the scientific
| process. Science is intended to be self-correcting.
| Replication is one means of this, but at its heart is the
| pursuit of counterfactual models. This gives us ways to
| validate, or invalidate, models through means other than
| direct replication. You can pursue the consequences of
| the results if you are unable to pursue the replication
| itself. This is almost always a good path to follow as it
| is the same one that leads to the extension and
| improvement of understanding.
|
| There's a lot I agree and disagree with from Dr Bissell's
| article. Our perspectives may differ due to our different
| fields, but I do think it also serves as some a point of
| collaboration, if not on the subject of meta-science.
| Biology is not unique in having expensive experiments. I
| want to point out two famous and large physics projects:
| the LHC's discovery of the Higgs Boson[0] and LIGO's
| Observation of a Gravitational Wave[1]. The former has 9
| full pages of authors (IIRC over 200) while the latter
| has about 3. These works are both too expensive to
| replicate while also demonstrating replication. Certainly
| we aren't going to take another 2 decades to build
| another CERN and replicate the experiments. But there's
| an easy to miss question that might also make apparent
| the existence of replication: who is qualified to review
| the paper and is not already an author of it? There's
| definitely some, but it really isn't that many. In these
| mega projects (and there are plenty more examples) the
| replication is done through collaboration. Independent
| teams examine the instruments that make the measurements.
| Independent teams make measurements, using the same
| device or different devices (ATLAS isn't the only
| detector at CERN), different teams independently analyze
| and process the information, and different teams model
| and simulate them. With LIGO this is also true. It would
| be impossible to locate those black holes without at
| least 2 facilities: one in Hanford (Washington) and the
| other in Livingston (Louisiana) (and now there's even
| more facilities). Astrophysics has a long history of this
| type of replication/collaboration as one team will
| announce an observation and it is a request for other
| observations. Observations that often were already made!
| In HEP (high energy particle physics) this may be less
| direct, but you'll notice other particle physics labs are
| in the author list of[0]. That's because despite the
| exact experiment not being replicatable in other
| facilities, there are still other experiments done. In
| the effort to find the Higgs there were many collisions
| performed at Fermi Lab.
|
| I don't think this same in biophysics, but I think there
| are nuggets that may be fruitful. Bissell mentions at the
| end of her argument that she believes replication might
| have higher success were labs to send scientists to the
| original labs. I fully agree! That would follow the
| practice we see in these mega experiments in physics. But
| I also do think she's brushing off an important factor:
| it is far quicker and cheaper to replicate works than it
| is to produce them. You're a scientist, you know how the
| vast majority of time (and usually the vast majority of
| money) is "wasted" in failures (it'd be naive to call it
| waste). Much of this goes away with replication efforts.
| The greater the collaboration the greater the reduction
| in time and money.
|
| And I do agree with Bissell in that we probably shouldn't
| replicate everything[2]. At least if we want to optimize
| our progress. But also I want to stress that there is no
| perfect system and there are many roadblocks to progress.
| Frankly, I'd argue that we waste far more time in things
| like grant writing and publication revisions. I don't
| know a single scientist who hasn't had a work rejected
| due to reviewers either not giving the work enough care
| or simply because they were unqualified (often working in
| a different niche so don't understand the minutia of the
| problem). As for the grant writings, I think they're a
| necessary evil but I'm also a firm believer of what
| Mervin Kelly (former director of Bell Labs) said when
| asked how you manage a bunch of geniuses: "you don't"[3].
| You're a scientist, an expert in your domain. You already
| know what directions to look in. You've only gotten this
| far because you've been honing that skill. We don't have
| infinite money, so of course we have to have some bar,
| but we can already sniff out promising directions and
| we're much better at sniffing out fraud. Science has been
| designed to be self-correcting.
|
| [More of a side note] > Usually the
| answer is, "if you're a good researcher and can't
| reproduce my work, you come to my lab and reproduce it
| there" because the variables that affect this are usually
| things in the lab- the temperature, the reagents, the
| handling.
|
| And we should not undermine the importance of these
| variables. Failures based on them are still informative.
| They still inform us about the underlying causal
| structure that leads to success. If these variables were
| not specified in the paper, then a replication failure
| shows the mistake of the writing. Alternatively a failure
| can bound these variables, by making them more explicit.
| I'm no expert in biophysics, but I'm fairly certain that
| understanding the bounds of the solution space is
| important for understanding how the processes actually
| work.
|
| [0] https://arxiv.org/abs/1207.7214
|
| [1] https://arxiv.org/abs/1602.03837
|
| [2] I also would be very cautious about paid replication
| efforts. I am strongly against it as well as paywalls on
| publishing (both in creation of publication as well as
| the access of).
|
| [3] https://1517.substack.com/p/why-bell-labs-worked
| godelski wrote:
| The problem is bigger. It even blocks research!
|
| In my own experience I was unable to publish a few works
| because I was unable to outperform a "competitor"
| (technically we're all on the same side, right?). So I
| dig more and more into their work and really try to
| replicate their work. I can't! Emailing the authors I get
| no further and only more questions. I submit the papers
| anyways, adding a section about replication efforts. You
| guessed it, rejected. With explicit comments from
| reviewers about lack of impact due to "competitor's"
| results.
|
| Is an experience I've found a lot of colleagues share.
| And I don't understand it. Every failed replication
| should teach us something new. Something about the bounds
| of where a method works.
|
| It's odd. In our strive for novelty we sure do turn down
| a lot of novel results. In our strive to reduce
| redundancy we sure do create a lot of redundancy.
| jltsiren wrote:
| I've seen this from both sides.
|
| Sometimes the result is wrong, or it's not as big or as
| general as claimed. Or maybe the provided instructions
| are insufficient to replicate the work. But sometimes the
| attempt to replicate a result fails, because the person
| doing it does not understand the topic well enough.
|
| Maybe they are just doing the wrong things, because their
| general understanding of the situation is incorrect.
| Maybe they fail to follow the instructions correctly,
| because they have subtle misunderstandings. Or maybe they
| are trying to replicate the result with data they
| consider similar, but which is actually different in an
| important way.
|
| The last one is often a particularly difficult situation
| to resolve. If you understand the topic well enough, you
| may be able to figure out how the data is different and
| what should be changed to replicate the result. But that
| requires access to the data. Very often, one side has the
| data and another side the understanding, but neither side
| has both.
|
| Then there is the question of time. Very often, the
| person trying to replicate the result has a deadline. If
| they haven't succeeded by then, they will abandon the
| attempt and move on. But the deadline may be so tight
| that the authors can't be reasonably expected to figure
| out the situation by then. Maybe if there is a simple
| answer, the authors can be expected to provide it. But if
| the issue looks complex, it may take months before they
| have sufficient time to investigate it. Or if the initial
| request is badly worded or shows a lack of understanding,
| it may not be worth dealing with. (Consider all the bad
| bug reports and support requests you have seen.)
| benterix wrote:
| > Do you want issues of Nature and cell to be replication
| studies?
|
| Of course I do! Not all of course, and taking (subjectively
| measured) impact into account. "We tried to replicate the
| study published in the same journal 3 years ago using a
| larger sample size and failed to achieve similar results..."
| OR "after successfully replicating the study we can confirm
| the therapeutic mechanism proposed by X actually works" -
| these are extremely important results that are takin into
| account in meta studies and e.g. form the base of policies
| worldwide.
| smj-edison wrote:
| Honestly even if they didn't publish the whole paper, if
| there was just a page that was a table of all the
| replication studies that were done recently, that would be
| pretty cool.
| paganel wrote:
| >Also who's funding you for replication work? Do you know the
| pressure you have in tenure track to have a consistent thesis
| on what you work on?
|
| This is partly why much of today's science is bs, pure and
| simple.
| notRobot wrote:
| "Original research" isn't worth much unless replicated, which
| is the entire problem being discussed in this thread.
| Replicating studies are great though because they tell you if
| the original research actually stands and is valid.
|
| > Replicating work is far more difficult than a lot of
| original work.
|
| Only if the original work was BS. And what, just because it's
| harder, we shouldn't do it?
| ramraj07 wrote:
| Why blame just the journals when every other system also
| disintivizes the same.
| anonymars wrote:
| I must be missing something, surely the argument isn't
| "other systems also disincentivize solving the problem,
| therefore we shouldn't work to fix this one"
| renewiltord wrote:
| Realistically, everyone will say "yes" to the "do you want"
| question because if you're not a reader or a subscriber you
| benefit from the readers reading replication studies.
|
| I believe people will enthusiastically say yes but that they
| do not routinely read that journal.
| andrewprock wrote:
| Suggesting that people would stop reading Nature if they
| also included replication studies send like an incredible
| leap.
| renewiltord wrote:
| That is a novel interpretation of my comment certainly.
| fc417fc802 wrote:
| It would directly undermine the reason that people read
| Nature in the first place.
| MarkusQ wrote:
| Not really.
|
| "It ain't what you don't know that gets you into trouble.
| It's what you know for sure that just ain't so."
|
| Knowing that something I thought was true was actually
| false would have saved me years in several situations.
| fc417fc802 wrote:
| I didn't understand us to only be talking about failed
| replication studies of previous Nature papers which would
| hopefully be few and far between and thus noteworthy
| indeed. Rather replication studies _in general_ which on
| average are arguably less interesting to the reader than
| even the content of the typical archival journal.
| MarkusQ wrote:
| They certainly will be few and far between when the
| system is structured to repress them. But there's reason
| to believe they wouldn't be as rare as you seem to think:
|
| https://www.nature.com/nature/articles?type=retraction
| fc417fc802 wrote:
| Are you seriously attempting to imply that Nature
| retractions aren't few and far between?
|
| What's even your point here? Hopefully we are at least in
| agreement that Nature is seen as prestigious and worth
| looking through precisely because of the sort of content
| that they publish. Diluting that would dilute their very
| nature. (Bad pun very much intended sorry I just couldn't
| resist.)
| MarkusQ wrote:
| "Are you seriously attempting to imply that Nature
| retractions aren't few and far between?"
|
| No. I'm explicitly stating that they are few and far
| between, but perhaps (not certainly, but conceivably)
| they shouldn't be.
|
| "What's even your point here?"
|
| My point is that focusing on positive findings and
| neglecting negative findings perverts the mechanism that
| makes science work. Science isn't about proving things
| correct, it's about rooting out errors.
| fc417fc802 wrote:
| I'm not sure I agree. The system certainly isn't optimal
| but results aren't just dumped into a vacuum. Something
| is only useful if people can build on it. Even if
| negative results don't get published, even if it isn't
| optimal, by virtue of future positive results building on
| past things that _did_ reproduce you get forward
| progress.
|
| Regardless, I don't think that's at odds with my original
| assertion that becoming a venue for publishing negative
| results would undermine the "point" of Nature.
|
| The missing link isn't a venue in which to publish. It's
| funding to do the work in the first place. Also funding
| to spend the time writing it up when you find that you've
| inadvertently been tricked into doing the work while
| trying to get something that builds on it to work.
| RobotToaster wrote:
| Tagging seems like an option here.
| chocochunks wrote:
| Even if that negative study could save you one, two, three+
| years of work for the same outcome (which you then also can't
| really do anything with)? Shouldn't there BE funding for
| replication studies? Shouldn't that count towards tenure?
| Part of the problem is that publications play such a heavy
| role in getting tenure in the first place.
|
| I'm sure you can more narrowly tune your email alerts FFS.
| peyton wrote:
| > Do you want issues of Nature and cell to be replication
| studies?
|
| Hell yeah. We're all trying to get that _Nature_ paper.
| Imagine if you could accomplish that by setting the record
| straight.
| fc417fc802 wrote:
| If you're thoroughly debunking a previous Nature paper they
| just might publish that. But the expectation is that you'll
| succeed. Publishing that sort of mundane article would
| reduce the prestige of getting something into the journal.
| Publishing in a high impact journal is only seen as an
| achievement in the first place because of what it implies
| about the content of your paper.
| xandrius wrote:
| I know you got a ton of responses already but not caring
| about replicability just invalidates science as a method. If
| we care only about first to publish we end up in the current
| situation where we don't even know that we know is actually
| even remotely correct.
|
| All because journals prefer novelty over confirmation. It's
| like a castle of cards, looks cool but not stable or long-
| term at all.
| lovich wrote:
| > Replicating work is far more difficult than a lot of
| original work.
|
| I don't regularly read scientific studies but I've read a few
| of them.
|
| How is it possible that a serious study is harder to
| replicate than it is to do originally. Are papers no longer
| including their process? Are we at the point where they are
| just saying "trust me bro" for how they achieved their
| results?
|
| > Do you want issues of Nature and cell to be replication
| studies?
|
| Not issues of Nature but I've long thought that universities
| or the government should fund a department of "I don't
| believe you" entirely focused on reproducing scientific
| results and seeing if they are real
| fc417fc802 wrote:
| > How is it possible that a serious study is harder to
| replicate than it is to do originally.
|
| They aren't. GP was on point until that last sentence. Just
| pretend that wasn't there. It's pretty much always much
| easier to do something when all the key details have been
| figured out for you in advance.
|
| There is some difficulty if something doesn't work to
| distinguish user error from ambiguity of original
| publication from outright fraud. That can be daunting. But
| the vast majority of the time it isn't fraud and simply
| emailing the original author will get you on track. Most
| authors are overjoyed to learn about someone using their
| work. If you want to be cynical about it, how else would
| you get your citation count up?
| carlosjobim wrote:
| If you're a reader within the field, then you are the one
| person in the world who should be most interested in negative
| replication studies.
| dheera wrote:
| Mainstream journals are complicit, but are not the biggest
| problem.
|
| The biggest problem by far is modern society: Tenure, getting
| paid a livable wage as a researcher, not getting stack-ranked
| and eliminated from your organization all overindex on positive
| research results that are marketable. This "loss function"
| encourages scientific fraud of sorts.
| bonoboTP wrote:
| When, in those mythical non-"modern" times, was it easy to
| get tenure or a livable wage as a researcher? How open were
| the doors to this and what proportion of society got a
| realistic chance to pursue such a career? More people getting
| a chance means more fierce competition.
| canjobear wrote:
| This isn't about honest researchers resorting to fraud to
| publish their null results because they were blocked by big bad
| Nature. It's about journals and authors churning out pure junk
| papers whose only goal is to game metrics like citation count.
| godelski wrote:
| > Most will refuse to publish replications, negative studies,
| or anything they deem unimportant, even if the study was
| conducted correctly.
|
| I think this was really caused by the rise of bureaucracy in
| academia. Bureaucrats favorite thing is a measurement,
| especially when they don't understand its meaning. There's
| always been a drive for novelty in academia, it's just at the
| very core of the game. But we placed far too much focus on
| this, despite the _foundation_ of science being replication. We
| made a trade, foundation for (the illusion of) progress. It 's
| like trying to build a skyscraper higher and higher without
| concern for the ground it stands on. Doesn't take a genius to
| tell you that building is going to come crashing down. But
| proponents say "it hasn't yet! If it was going to fall it would
| have already" while critics are actually saying "we can't tell
| you when it'll fall, but there's some concerning cracks and
| we're worried it'll collapse and we won't even be able to tell
| we're in a pile of rubble."
|
| I don't know what the solution is, but I do know that our fear
| of people wasting money and creating fraudulent studies has
| only resulted in wasting money and fraudulent studies. We've
| removed the verification system while creating strong
| incentives to cheat (punish or perish, right?).
|
| I think one thing we do need to recognize is that in the grand
| scheme of things, academia isn't very expensive. A small
| percentage of a large number is still a large number. Even if
| half of academics were frauds it would be a small percentage of
| waste, and pale in comparison to more common waste, fraud, and
| abuse of government funds.
|
| From what I can tell, the US spent $60bn for University R&D in
| 2023[0] (less than 1% of US Federal expenditures). But in that
| same time there was $400bn in waste and fraud through Covid
| relief funds [1]. With $280bn being straight up fraud. That
| alone is more than 4x of all academic research funding!!!
|
| I'm unconvinced most in academia are motivated by money or
| prestige, as it's a terrible way to achieve those things. But I
| am convinced people are likely to commit fraud when their
| livelihoods are at stake or when they can believe that a small
| lie now will allow them to continue doing their work. So as I
| see it, the publish or perish paradigm only promotes the
| former. The lack of replication only allows, and even
| normalizes, the latter. The stress for novelty only makes
| academics try to write more like business people, trying to
| sell their product in some perverse rat race.
|
| So I think we have to be a bit honest here. Even if we were to
| naively make this space essentially unregulated it _couldn 't_
| be the pinnacle of waste, fraud, and abuse that many claim it
| is. But I doubt even letting scientists be entirely free from
| publication requirements that you'd find much waste, fraud, and
| abuse. Science has a naturally regulating structure. It was
| literally created to be that way! We got to where we are in
| through this self regulating system because scientists love to
| argue about who is right and the process of science is meant to
| do exactly that. Was there waste and fraud in the past? Yes. I
| don't think it's entirely avoidable, it'll never be $0 of waste
| money. But the system was undoubtably successful. And those
| that took advantage of the system were better at fooling the
| public than they were their fellow scientists. Which is
| something I think we've still failed to catch onto
|
| [0] https://usafacts.org/articles/what-do-universities-do-
| with-t...
|
| [1] https://apnews.com/article/pandemic-fraud-waste-billions-
| sma...
| bonoboTP wrote:
| You either have something documented and quantified and
| measured and objective criteria tickboxes and deal with this
| style of failure mode, or you rely on subjective judgment and
| assessment and accept the failure mode of bias, nepotism, old
| boy's clubs etc. Of course the ideal case is to rely on the
| unbureaucratic informal wise and impartial judgment of some
| hypothetical perfect humans you can fully trust and rely on,
| and they always decide fully on merits etc. without having to
| follow any rigid criteria and checkboxes and numbers on
| hiring and promotion etc. But people are not perfect and
| society largely decided to go the bureaucratic way to ensure
| equal opportunities and to reduce bias through this kind of
| transparency.
| godelski wrote:
| > You either have something documented and quantified and
| measured and objective criteria tickboxes and deal with
| this style of failure mode, or you rely on subjective
| judgment and assessment and accept the failure mode of
| bias, nepotism, old boy's clubs etc
|
| My argument is that our current pursuit of the former only
| reinforces the existence of the latter.
|
| You have a fundamental flaw in your argument, one that
| illustrates a common, yet fundamental, misunderstanding of
| science. There is no "objective" thing to measure, there
| are only proxies. I actually recently stumbled on a short
| by Adam Savage that I think captures this[0], although I
| think he's a bit wrong too. Regardless of precision we are
| always using a proxy. A tape measure does not define a
| meter, it only serves as a reference to compare with. A
| reference where not only the human makes error when
| reading, but that the reference itself has error[1]. So
| there are no direct measurements, there are only
| measurements by proxy.
|
| You may have heard someone say "science doesn't prove
| things, it disproves them", and that's in part a
| consequence to this. Our measurements are meaningless
| without an understanding of their uncertainty (both
| quantifiable and unquantifiable!) as well as the
| assumptions they are made under.
|
| I'm not trying to be pedantic here, I think this precision
| in understanding matters to the conversation. My argument
| is that by discounting those errors that they accumulate.
| We've had a pretty good run. This current system has only
| really started to be practiced in the 60s and 70's. So 50
| years is a lot of time for error to accumulate. 50 years is
| a lot of time for small, seemingly insignificant, and easy
| to dismiss errors to accumulate into large, intangible, and
| complex problems.
|
| There's something that I guess is more subtle in my
| argument: science is self-correcting. I don't mean
| "science" as the category of pursuits that seek truths
| about the world around us, but I mean "science" as a
| systematic approach to obtaining knowledge. A key reason
| this self-correction happens is due to replication. But in
| reality that is a consequence of how we pin down truth
| itself. We seek causal structures. More specifically, we
| seek counterfactual models. Assuming honest practitioners,
| failures of reproduction happen for primarily for one of
| two reasons: 1) ambiguity of communication between the
| original experimenters and those replicating or 2) a
| variation in conditions. 2) is actually quite common and
| tells us something new about that causal structure. In
| practice it is extremely difficult, if not impossible, to
| exactly replicate the conditions of the original
| experiment, so even with successful replication we gain
| information about the robustness of the results.
|
| But why am I talking about all this? Because without the
| explicit acknowledgement of these limitations we seem to
| easily forget them. We are often treating substantially
| more subjective measures (such as impact or novelty) as far
| more objective than we would treat even physical
| measurements. It should be absolutely no surprise that
| things like impact are at best extremely difficult to
| measure. Even with a time machine we may not accurately
| measure the impact of a work for decades, or more.
| Ironically, a major reason for a work's impact to be found
| only after decades (or centuries) is the belief that at its
| time it had no impact, and was a dead end. You'd be amazed
| at how common this actually is. It's where jokes similar to
| how everything is named after the second person to discover
| something, the first being Euler[2]. But science is self-
| correcting. Even if a discovery of Euler's was lost, it is
| only a matter of time before someone (independently)
| rediscovers it.
|
| I'm talking about this because there is no perfect system.
| Because a measurement without the acknowledge of its
| uncertainty is far less accurate than a measurement with.
| I'm talking about this because we will always have errors
| and the existence of them is not a reason to dismiss
| things. Instead we have to compare and contrast both the
| benefits and limits of competing ideas. We are only doing
| ourselves a disservice by pretending the limits don't
| exist. And if we mindlessly pursue objective measurements
| we'll only end up finding we've metric hacked our way into
| reading tea leaves. As we advance in any subject the
| minutia always ends up being the critical element (see [0])
| and so the problem is it doesn't matter if we're 90%
| "objective" and 10% reading the tea leaves. Not when the
| decisions are made differentiating the 10%. In reality
| we're not even good at measuring that 90% when it comes to
| determining how productive academics are[3-5]
|
| [0] https://www.youtube.com/shorts/JGa_X4QfE-0
|
| [1] https://www.youtube.com/watch?v=EstiCb1gA3U
|
| [2] https://en.wikipedia.org/wiki/List_of_topics_named_afte
| r_Leo...
|
| [3] https://briankeating.substack.com/p/peter-higgs-
| wouldnt-get-...
|
| [4] https://yoshuabengio.org/2020/02/26/time-to-rethink-
| the-publ...
|
| [5] See the two links in this comment as further evidence.
| They are about relatively recent Nobel works that faced
| frequent rejections
| https://news.ycombinator.com/item?id=47340733
| bonoboTP wrote:
| Someone has to pay for all this. That someone is most
| often not a scientist themselves. They don't have a that
| vague intuitive research taste that scientists have.
| Beyond fairly trivial levels of technical correctness,
| the value of research lies in its narrative implications,
| its interestingness, its surprise factor etc. These are
| not objective and are often more about aesthetics and
| taste than popularly understood. Why does the research
| matter? To whom does it matter? Are those people
| important? Do they control resources?
| bonoboTP wrote:
| Plenty will publish it, but those are not as highly regarded by
| the community. It's not a problem of journals. It's not hard to
| start your own journal by teaming up with other academics. In
| machine learning, ICLR is such a venue for example. The problem
| is much deeper and more fundamental. You want to publish
| alongside groundbreaking novel research. Researcher's own ears
| perk up when they hear about something new. They invite
| colleagues to talk about their novel discoveries not to
| describe all their null results and successful replications of
| known results. Funding agencies want research with novelty and
| impact. They want to write reports to the higher ups and the
| politicians and the donors that document the innovations that
| their funding brought. The media will republish press releases
| that have cool new results.
|
| To have research happening, you need someone saying "I want to
| give money to this researcher". There is an endless queue of
| people lining up who are ready to take this money and do
| something with it. The person with money (govt or private) has
| to use some heuristics to pick. One way is to say "I trust this
| one, I don't care too much what the project is, I'm sure this
| person will do something that makes sense". But that is
| dependent on a track record.
| obviouslynotme wrote:
| I have worked in this particular sausage factory. Multiple
| funded random replications are the only thing that will save
| science from this crisis. The scientific method works. We need
| to actually do it.
|
| Replications don't have to be in the journals either. As long
| as money flows, someone will do them, and that is what matters.
| The randomization will help prevent coordination between
| authors and replicators.
|
| In a better world, negative studies and replications would
| count towards tenure, but that is unlikely to occur. At least
| half of the problem is the pressure to continuously publish
| positive results.
| bonoboTP wrote:
| Regardless of what gets taught in school about science being
| objective and without ego, or having a culture of adversarial
| checks on each other etc., the reality is that scientists are
| humans and have egos and have petty feuds.
|
| Publishing a failed replication of the work of a colleague
| will not earn you many brownie points. I'm stating this as an
| observation of what is the case, not as something that I
| think should be the case. If you attack other researchers
| like this and damage their reputation - even if for valid
| scientific reason - you'll have a hard time when those
| colleagues sit on committees deciding about your next grant
| etc.
|
| Of course if you discover something truly monumental that
| will override this. But simply sniping down the mediocre
| research published by other run-of-the-mill researchers will
| get you more trouble than good. Yes it's directly in
| contradiction to the textbook-ideal of what science should
| be, as described to high school students, but there are many
| things in life this way.
|
| Of course it can be laudable to go on such a crusade despite
| all this, and to relentlessly pursue scientific truth, etc.
| but that just won't scale.
| obviouslynotme wrote:
| You are absolutely correct. Even distributing the
| replication around the world will only help so much. It's a
| small world out there and only smaller in the
| specializations.
|
| That's why replication has to be required and standard. It
| will hurt to tear off the bandaid, but once the culture
| shifts, people will hesitate to publish mediocre research
| in the first place. Without mediocre research flooding the
| zone, real numbers will dominate and inflated expectations
| will wither.
| temporallobe wrote:
| My wife completed her PhD two years ago and she put a LOT of work
| into it. Many sleepless nights, and it almost destroyed our
| marriage. It took her about 6 years of non-stop madness and she
| didn't even work during that time. She said that many of her
| colleagues engaged in fraudulent data generation and sometimes
| just complete forgery of anything and everything. It was obvious
| some people were barely capable of putting together coherent
| sentences in posts, but somehow they generated a perfect
| dissertation in the end. It was common knowledge that candidates
| often hired writers and even experts like statisticians to do
| most of the heavy lifting. I don't know if this is the norm now,
| but I simultaneously have more respect and less respect for those
| doctoral degrees, knowing that some poured their heart and soul
| into it, while others essentially cheated their way through.
| OTOH, I also understand that there may be a lot of grey area.
|
| My eyes have been opened!
| titzer wrote:
| I found the article and your third-hand anecdotes troubling.
| The good news is that it does not match _any_ of the years of
| experience in my field. Fraud is just not that rampant. At PhD-
| granting institutions, the level of fraud you describe here is
| very seriously punished. It 's career-ending. The violations
| that you are serious enough that any institution would expel
| said students (or harshly punish faculty--probably firing
| them). She did no one any favors by not reporting them.
|
| Unfortunately I don't think a dialogue around vague anecdotes
| is going to be particularly enlightening. What matters is
| culture, but also process--mechanisms and checks--plus
| consequences. Consequences don't happen if everyone is hush-
| hush about it and no one wants to be a "rat".
| qsera wrote:
| >It's career-ending..
|
| That is where being good at politics come into play. And if
| you are good at it, instead of being career-ending, fraud
| will put you in the highest of the positions!
|
| No one wants a "plant" who cannot navigate scrutiny!
| delichon wrote:
| > The good news is that it does not match any of the years of
| experience in my field.
|
| I worked for exactly one academic, and he indulged in
| impossible-to-detect research fraud. So in my own limited
| experience research fraud was 100%.
|
| It was a biology lab, and this was an extremely hard working
| man. 18 hours per day in the lab was the norm. But the data
| wasn't coming out the way he wanted, and his career was at
| stake, so he put his thumb on the scale in various ways to
| get the data he needed. E.g. he didn't like one neural
| recording, so he repeated it until he got what he wanted and
| ignored the others. You would have to be right in the middle
| of the experiment to notice anything, and he just waved me
| off when I did.
|
| This same professor was the loudest voice in the department
| when it came to critiquing experimental designs and
| championing rigor. I knew what he did was wrong, because he
| taught me that. And he really appeared to mean it, but when
| push came to shove, he fiddled, and was probably even lying
| to himself.
|
| So I came away feeling that academic fraud is probably
| rampant, because the incentives all align that way. Anyone
| with the extraordinary integrity to resist was generally
| self-curated out of the job.
| dekhn wrote:
| I had a somewhat similar experience- was a postdoc for a
| pre-tenure professor at berkeley. after writing up a paper
| based on her methods, with poor results, I handed the draft
| to her. She rewrote it- basically adding carefully
| worded/presented results that made it look as good as
| possible. And then submitted it (to a niche conference
| where the editor was a buddy of hers). When I read her
| submission I asked her to remove my name from it and she
| immediately withdrew the submission. I left her lab shortly
| after because I am not going to tarnish my publication
| record with iffy papers like that.
|
| Over time I learned that most papers in my field
| (computational biology) are embellished to some extent or
| another (or cherry-picked/curated/structured for success)
| and often irreproducible- some key step is left out, or no
| code is provided that replicates the results, etc. I can
| see this from two perspectives:
|
| 1) science should be trivially reproducible; it should not
| require the smartest/most capable people in the field to
| read the paper and reproduce the results. This places a
| burden on the people who are at the state of the art of the
| field to make it easy for other folks, which slows them
| down (but presumably makes overall progress go faster).
|
| 2) science should be done by geniuses; the leaders in the
| field don't need to replicate their competitors paper. it's
| sufficient to read the paper, apply priors, and move on
| (possibly learning whatever novel method/technique the
| paper shows so they can apply it in their own hands). It
| allows the field innovators to move quickly and discover
| new things, but is prone to all sorts of
| reliability/reproducibility problems, and ideally science
| should be egalitarian, not credentials-based.
| renewiltord wrote:
| My cousin (with whom I am very close) had a similar
| experience that I posted about years ago.
| https://news.ycombinator.com/item?id=32969092
|
| I have repeated it many times on this site but here's the
| reality of human experience: if the rate of fraudulent labs
| is even as high as 10% you should expect that any viewpoint
| that it's widespread would be drowned out by views that
| it's not real.
|
| Also, the phenomenon you observed where people are
| champions till the rubber meets the road is more common
| than one thinks.
| Noumenon72 wrote:
| > if the rate of fraudulent labs is even as high as 10%
| you should expect that any viewpoint that it's widespread
| would be drowned out by views that it's not real.
|
| If "it" is fraud here I would expect the viewpoint that
| it's widespread to be less and less drowned out as it
| approached 10% since everyone would know that it's real.
| I think I'm misunderstanding the sentence.
| renewiltord wrote:
| No, the guys at fraudulent labs and the guys at honest
| labs will both claim no fraud. The only ones who will
| claim fraud are those who cross over. So you'll get a
| vast majority telling you it's not happening and a tiny
| minority (even when as high as 10% are fraud) telling you
| the fact. All rare things have this effect. There will be
| so many people telling you it's not real "as someone in
| the field". They will be adamant about it. You need
| someone who has seen both.
|
| To be clear, not "as it approaches 10%". I mean "even as
| high as 10%".
| mistrial9 wrote:
| yeah - skeptical here. Among certain departments, at large
| schools, under certain leaders.. The combination of "my
| marriage almost crumbled" for motivated reasoning, and "I have
| never seen any of this before" total inexperience with actual
| process.. the post shows itself to be biased and unreliable.
|
| However, among certain departments, at large schools, under
| certain leaders.. yes, and growing
|
| $0.02
| russdill wrote:
| Fucking hilarious to me when people claim academics are
| motivated by the "money", eg, when claimed by climate deniers.
| 1234letshaveatw wrote:
| Undoubtably climate science is the exception and immune from
| fraudulent data generation and sometimes complete forgery
| russdill wrote:
| I'm sure there's some, but the small point here is that it
| almost certainly is more motivated by factors other than
| financial gain. I'm sure it you search you can find such
| cases though.
|
| The much broader point though is the dismissal of the bulk
| consensus of academic research because academics are in it
| for the "money".
| suddenlybananas wrote:
| What field? I am aware this kind of stuff happens, but I don't
| really see it among any of my colleagues.
| fastaguy88 wrote:
| It is useful to distinguish between "effective" scientific fraud,
| where some set of fraudulent papers are published that drive a
| discipline in an unproductive direction, and "administrative"
| scientific fraud, where individuals use pseudo-scientific
| measures (H-index, rankings, etc) to make allocation decisions
| (grants, tenure, etc). This article suggests that administrative
| scientific fraud has become more accessible, but it is very
| unclear whether this is having a major impact on science as it is
| practiced.
|
| Non-scientists often seem to think that if a paper is published,
| it is likely to be true. Most practicing scientists are much more
| skeptical. When I read a that paper sounds interesting in a high
| impact journal, I am constantly trying to figure out whether I
| should believe it. If it goes against a vast amount of science
| (e.g. bacteria that use arsenic rather than phosphorus in their
| DNA), I don't believe it (and can think of lots of ways to show
| that it is wrong). In lower impact journals, papers make claims
| that are not very surprising, so if they are fraudulent in some
| way, I don't care.
|
| Science has to be reproducible, but more importantly, it must be
| possible to build on a set of results to extend them. Some
| results are hard to reproduce because the methods are technically
| challenging. But if results cannot be extended, they have little
| effect. Science really is self-correcting, and correction happens
| faster for results that matter. Not all fraud has the same
| impact. Most fraud is unfortunate, and should be reduced, but has
| a short lived impact.
| qsera wrote:
| >methods are technically challenging.
|
| And finanacially too..
|
| >Science really is self-correcting..
|
| When economy allows it....
| perfmode wrote:
| The distinction between effective and administrative fraud is
| useful and I think underappreciated. A lot of the conversation
| in these threads conflates the two, which makes it hard to
| reason about what actually needs fixing.
|
| I want to push back a little on "science is self-correcting"
| though. It's true in the limit, but correction has a latency,
| and that latency has real costs. In fields like nutrition,
| psychology, or pharmacology, a fraudulent or deeply flawed
| result can shape clinical guidelines, public policy, and drug
| development pipelines for a decade or more before the
| correction lands. The people harmed during that window don't
| get made whole by the eventual retraction.
|
| The comparison I keep coming back to is fault tolerance in
| distributed systems. You can build a system that's "eventually
| consistent" and still have it be practically broken if
| convergence takes too long or if bad state propagates faster
| than corrections do. The fraud networks described in TFA are
| basically an adversarial workload against a system (peer
| review) that was designed for a much lower rate of bad input.
| Saying the system self-corrects is accurate, but it's not the
| same as saying the system is healthy or that the current
| correction rate is adequate.
|
| I think the practical question isn't whether science corrects
| itself in theory but whether the feedback loops are fast enough
| relative to the rate of fraud production, and right now the
| answer seems pretty clearly no.
| pfdietz wrote:
| One approach is more integration of researchers with businesses.
| Fraud (or simple incompetence) by researchers negatively affects
| businesses, as they expend effort on things that aren't real. I
| understand this is a constant problem in the pharmaceutical
| industry.
| robmccoll wrote:
| It's quite possible to be very successful marketing and selling
| things that aren't real. The market consists of humans, not
| perfectly rational machines.
| stanford_labrat wrote:
| the problem is two-fold in my opinion.
|
| firstly, there are basically no _legal_ repercussions for
| scientific misconduct (e.g. falsifying data, fake images, etc.).
| most individuals who are caught doing this get either 1) a slap
| on the wrist if they are too big to fail or in the employ of
| those who are too big to fail or 2) disbarred, banned, and lose
| their jobs. i don 't see why you can go to jail for lying to
| investors about the number of users in your app but don't go to
| jail for lying to the public, government, and members of the
| scientific community about your results.
|
| secondly, due to the over production of PhD's and limited number
| of professorship slots competition has become so incredibly
| intense that in order to even be considered for these jobs you
| _must_ have Nature, Cell, and Science papers (or the field
| equivalent). for those desperate for the job their academic
| career is over either way if they caught falsifying data or if
| they don 't get the professorship. so if your project is not
| going the way you want it to then...
|
| sad state of things all around. i've personally witnessed enough
| misconduct that i have made the decision to leave the field
| entirely and go do something else.
| noslenwerdna wrote:
| I unironically agree, p-hacking should be a criminal offense.
| pjdesno wrote:
| Perhaps relevant to this - if you go to this global ranking of
| publications:
| https://traditional.leidenranking.com/ranking/2025/list
|
| and select "Mathematics and Computer Science", you'll find the
| top-ranked university is the University of Electronic Science and
| Technology of China.
|
| My Chinese colleagues have heard of it, but never considered it a
| top-ranked school, and a quick inspection of their CS faculty
| pages shows a distinct lack of PhDs from top-ranked Chinese or US
| schools. It's possible their math faculty is amazing, but I think
| it's more likely that something underhanded is going on...
| Atlas667 wrote:
| Almost as if capitalism makes everything into a market, and the
| profits make it self sustaining.
|
| How many will see the connections between this and our capitalist
| mode of production? Probably few since modern lit/news is
| allergic to systemic analysis.
|
| The blatant flaws of capitalism can't be ignored for much longer.
| pooooka wrote:
| What I get from this is that the professional academic
| community -- as a whole -- has hit critical mass, which has
| produced a cottage industry of paper mills and fraudulent
| services to support said surplus.
|
| Socialism wouldn't be the answer to this because socialism is
| famous for struggling with surpluses and shortages. All
| socialism would do is clamp down (hard) on academic's, which
| case you wind up with the famous shortage where not enough
| PHD's are available to produce research for an industry.
|
| And that's not a problem specific to just socialism, that's the
| fallacy of central-planning. The US government clamped down on
| welfare fraud and the result were freak government social
| workers sniffing people's bed sheets and rooting through
| drawers and forcing everyone to document partners.
|
| This is the situation where there needs to be a market
| correction because the alternative could be far worse.
| Atlas667 wrote:
| It's the tax-payer funded business model, the NGO trap.
| Subsidies, grants, tax-breaks, credit, deductions,
| exemptions, etc. A whole class of profiteers live in this
| sector. Even though academia funding isn't strictly
| categorized as an NGO, it still fits/foots the bill. Public
| funding of private gains is the oldest trick in the book. Ask
| any capitalist, they know. And I'm not saying I'm against
| public funding, but this is often codified into a mafia of
| sorts when enough money flows through.
|
| The real problem here is the fundamental lack of democratic
| control over our agencies. That our political organization is
| intensely lagging behind our productive organization. That
| our whole political will involves TRUSTING strangers to not
| be corrupt instead of directly democratizing these processes
| as much as possible.
|
| But besides that, you cannot remove history from historical
| analysis. The reason socialism countries struggled in the
| beginning wasn't an inherent flaw in its organization, but
| the fact that they were under constant war war by capitalist
| countries through out their existence. Also keep in mind that
| most socialist countries did NOT have a whole section of the
| world where-from to extract riches through murder (S.America,
| Africa, Middle east, etc), like western capitalist countries
| had. This is convenient for you to ignore. Maybe because you
| don't know, or don't care about the super-exploitative
| history of these places and how they tie into western
| capitalism. But they are inherent to western wealth and these
| countries' whole history is struggle against this
| exploitation.
|
| Not to mention that most of the countries on earth are
| capitalists and are very very very poor.
|
| To add: Socialism has nothing to do with "clamping down" on X
| or Y industry, as you hypothetically claim would happen.
| Socialism is almost exclusively about removing the need to
| generate capital from production. It unleashes production
| from its historical ball and chain that is profiteering.
|
| In a single sentence: Instead of production being held back
| by capitalists generating wealth we can produce for our own
| needs. It is self sustaining production.
|
| Central planning is not fallacious. Your problem is with
| corruption, not democratic central planning. The US Govt is a
| pro-capitalist entity that pro-capitalists try to distance
| themselves from (ironically). So using them as an example
| isn't saying anything at all.
|
| Central planning is not "allow a small group of people to
| decide things", as happens in the US Govt. Central planning
| is to take into account all sources of information on
| production to plan said production democratically.
|
| This will always beat the highly highly inefficient
| speculation of capitalism. Where trillions vanish on a whim
| and cause of a tweet, where crisis occur every 8-10 years,
| and where its whole trade market is built to hide that it is
| mostly insider trading. Again, your problem is with
| corruption not democratic central planning.
|
| And the way to deal with corruption is to create more
| democratic bodies where avg people hold real power. I don't
| see you asking for that either. We call that socialism.
| orbital-decay wrote:
| All people in my extended family were Soviet scientists and
| engineers from multiple fields, and outside of experimental
| physics it was the same or worse. Same publish or perish
| pressure, same amount of fraud and lack of reproducibility. A
| ton of papers were made up. My father's lab lead was an
| absolute fraud (biochemistry), everybody knew that, and my
| father was unable to speak up until the late 90's.
|
| When I was a kid I thought it was the issue with USSR rotting
| to the core (it was), but when it crashed and later when the
| web appeared, it became obvious that it's a common problem with
| academia and its incentives.
| pmarreck wrote:
| why would anyone actually interested in scientific research come
| to this, since it literally undermines the whole practice of
| science?
| cyberjerkXX wrote:
| Publish or perish. Academia requiring PhDs to publish or be
| fired. It's made entire fields echo chambers and prone to
| political influence.
| pmarreck wrote:
| so a perverse incentive, basically. shocker. got it
| zahlman wrote:
| It's strange to me that in places full of smart people, it seems
| to be well understood that this happens and there are lots of
| anecdotes relating to it; yet the same people will be confused
| that their political adversaries don't trust "the science" on one
| issue or another.
|
| Maybe it's the scien _tists_ they don 't trust?
| Hendrikto wrote:
| That's the beautiful thing about science: You do not have to
| (and should not) trust any individual. And even if you don't
| trust "the consensus" of "the scientific community", you can
| empirically verify yourself.
| zahlman wrote:
| Can ordinary civilians feasibly measure, for example, global
| trends in mean temperature without relying on the data of
| others?
| fc417fc802 wrote:
| No, but the literature is open for you to read. Thus you
| can judge the stated reasoning for yourself. You can also
| assess how many independent groups are making the same (or
| closely related) claim.
|
| If only one person claims X then it might be fraud. If
| large numbers of seemingly unrelated people all claim X
| then you're forced to decide between X and a global
| conspiracy to misrepresent X.
|
| To your example. Importantly, even if you deemed one of the
| global mean temperature datasets to be untrustworthy there
| are other related (but different) datasets. There are also
| other pieces of evidence related to the downstream claims
| that don't look directly at temperature.
| dekhn wrote:
| Are you going to build a competitor to CERN?
|
| There are many things that cannot be feasibly verified
| empirically without access to rare resources.
| tbrownaw wrote:
| Once you move from abstract to practical - like say having
| legislators or regulators make rules based on The Science, or
| relying personally on more facts than you have time to
| independently verify - yes you do need to have trustworthy
| people.
| bonoboTP wrote:
| I think it's difficult to relay to the public that a lot of
| this noise in "scientific publications" is not the same
| category as real research by reputable institutions. Yes, in
| certain cases the line can be blurry, fraudsters are sometimes
| caught in big-name institutions, maybe more in some fields than
| others, but serious researchers of the field know very well
| which publication venues and research groups are the real deal
| and what is bullshit. Overwhelmingly, these fraud papers and
| nonsense LLM-generated fake stuff are not published in serious
| journals or conferences.
|
| It's a bit like how can we trust online shopping if I get all
| these emails trying to sell me aphrodisiac pills?
| gadders wrote:
| This is what happens when people argue past each other on "Trust
| the science".
|
| Science is good, but it's mediated via corruptible humans.
| MarkusQ wrote:
| Also, "science" isn't some sort of dogma that you should trust,
| it's a process you should follow.
|
| "Trust the science" is anathema to the process. If anything,
| the chant should be "Doubt the science! Give it your best shot,
| refute it with data, with logic, provide a better explanation!"
| bonoboTP wrote:
| Realistically, the vast majority of people will not have a
| real chance to "refute" or even evaluate scientific claims.
| Maybe given a lot of time and foundational work to learn the
| field, some percentage of people can usefully think about
| them, but the vast majority can't. A lot of people are
| functional illiterates. They will pick based on trust and gut
| feelings either way.
|
| For example, when deciding whether to give your kids certain
| vaccines or not, you really can't expect that new parents
| will read the primary literature and try to refute or confirm
| the conclusions based on the numbers and will trace through
| the citations and so on... Any of those claims will also have
| some online account on social media refuting it with equally
| scientifically sounding words. In the end it will come down
| to heuristics and your model of how the world works, which
| set of people operate with what kind of intention. Like maybe
| you know people working in the field who you trust and hear
| from them that generally this sort of stuff can be trusted.
| Or maybe you had some bad experiences getting screwed by "the
| establishment" (maybe even unrelated to medicine) and now you
| lump all this together and distrust them.
| barbazoo wrote:
| It always comes back to Goodhart's Law and our apparent inability
| to create sustainable incentive structures.
| butILoveLife wrote:
| Industry >> Academia
|
| Profits are the deciding factor, not honor.
| jjk166 wrote:
| More broadly, an incredible amount of our society's systems are
| built around actors being uncoordinated. Redesigning institutions
| to resist networks of coordinated action between seemingly
| unlinked individuals will, in my opinion, be one of the great
| social challenges of this era.
| ukoki wrote:
| If you get paid by the government to do research you should make
| all your raw data, code, results etc, accessible to the public.
|
| If it then turns out any of it is fabricated, you should be
| personally liable for paying it back
| canjobear wrote:
| I ran into an interesting incident of this recently. I got a
| Google Scholar alert about a paper with some experiments related
| to a paper I had published a while ago, by one "N. Tvlg". I read
| the paper with interest but I started noticing that although the
| arguments sounded good, they didn't really make sense, and also
| the descriptions of the results didn't really match the figures.
| Eventually I came across a cluster of citations to completely
| unrelated papers---my field is computational linguistics and
| these were citations to, like, studies of battery technologies
| for electric cars. I looked up "N Tvlg" on Google Scholar and
| they had "published" several articles very recently in totally
| divergent fields, and upon inspection, all of them had citations
| back to this materials science research buried deeply somewhere.
| Clearly these were LLM generated papers trying to build up
| citation count and h-rank for someone's career.
| reactordev wrote:
| Where there's a ranking, there's someone out there trying to
| cheat at it. Citation count is a joke.
| matthewdgreen wrote:
| The purpose of scientific publication used to be to deliver
| useful scientific results to one's peers. This meant that
| everyone ran their own personal filter of which peers were
| working on interesting things, and which collections (journals)
| were reproducing the most interesting ones. This system still
| works relatively well for most conscientious researchers. The
| idea that we should also use publication metrics to rank
| researchers was never part of this system, and it obviously
| leads to all sorts of spam (that most scientists just work
| around) but that seems to really upset non-scientists.
| fph wrote:
| Are these "entities" named and shamed somewhere? I just scanned
| the paper but couldn't find explicit mentions.
| ventuss_ovo wrote:
| This is the part that feels hardest to fix: once a system starts
| rewarding throughput over scrutiny, fraud stops looking like
| individual misconduct and starts looking like a supply chain
| problem.
___________________________________________________________________
(page generated 2026-03-11 23:00 UTC)