[HN Gopher] Transparent peer review to be extended to all of Nat...
___________________________________________________________________
Transparent peer review to be extended to all of Nature's research
papers
Author : rntn
Score : 99 points
Date : 2025-06-16 18:51 UTC (4 hours ago)
(HTM) web link (www.nature.com)
(TXT) w3m dump (www.nature.com)
| jostmey wrote:
| This only addresses a small part of the problem with peer-review.
| The real problem is that peer reviewers can't possibly replicate
| the study, and so are forced to look for inconsistencies in the
| papers. If the paper doesn't fit what is expected, it will often
| be rejected. This can also lead to self-reinforcing views that
| ignore contrarian data. Also, the data can be made up, and if it
| makes sense to the reviewers, it is generally not questioned
| jxjnskkzxxhx wrote:
| Reading your comment makes me think that you believe that the
| point of peer-review is to ensure that a paper is correct, or
| at least that specific aspects of it are correct. Is that the
| case? What do you think the point of peer-review is?
| yupitsme123 wrote:
| I'm not the person you replied to, but I think that in the
| lay world, people do indeed think that peer review is as
| you've described. If it's not, then maybe it should be?
|
| Research gets cited constantly in public debates and is used
| for policy decisions, so the public should be able to quickly
| separate the good from the bad, the "maybe this is true" from
| the "this is empirically proven."
|
| The public has lost a lot of trust in Science because
| research papers have been used to push political agendas,
| which can then never be questioned because doing so means
| arguing with a supposed peer-reviewed scientific consensus.
| thomasfedb wrote:
| Nothing is ever "proven". There is simply more or less
| support for a theory or proposition.
|
| Replication and meta-analysis are an important part of
| this.
|
| Most scientists are in fact very conservative with how they
| claim their results - less so university PR departments and
| "study shows" clickbaiters.
| jxjnskkzxxhx wrote:
| I approve of this comment.
| JumpCrisscross wrote:
| > _public has lost a lot of trust in Science because
| research papers have been used to push political agendas,
| which can then never be questioned because doing so means
| arguing with a supposed peer-reviewed scientific consensus_
|
| The public has lost trust in science because 10 to 30% of
| it is scientifically illiterate [1]. (Tens of millions of
| American adults are literally illiterate [2].)
|
| That's what lets activists and politicians cherrypick bad
| science that supports their position or cast a scientific
| consensus as unquestionable.
|
| [1] https://carsey.unh.edu/publication/conspiracy-vs-
| science-sur...
|
| [2] https://nces.ed.gov/pubs2019/2019179/index.asp
| fc417fc802 wrote:
| That's certainly true but I think there's also a very
| real issue as described by GP. Research gets cited in a
| political setting as a rhetorical tool. That cycle does a
| lot to erode trust in the establishment because it
| incentivizes non-scientists (who are otherwise
| uninvolved) to behave as though the process is a partisan
| effort to be interfered with for the benefit of one's
| "team" rather than an objective pursuit to be funded at
| arms length for the betterment of society at large.
|
| Obviously reproducing results as part of peer review is
| not a workable (or even coherent) solution. I don't
| pretend to have any idea what a solution might be. The
| obvious issue is that academic publications were never
| intended as political tools and should not be made into
| that.
|
| On several occasions I've had interactions with laymen
| where I found myself thinking "if only you hadn't had
| access to pubmed and way too much motivation we'd both be
| better off right now" yet I firmly believe that free and
| open access to information is a huge net benefit to
| society on the whole.
| aDyslecticCrow wrote:
| Bad quality work is a much bigger problem than dishonest work.
| Systematically well-done research with fake results or data is
| much rarer than just... lazy bad science.
|
| There is a massive incentive to publish. Inflate the value,
| inflate the results, and stretch out projects to multiple
| smaller papers, fake results to make it seem important. This is
| lazy and fast, and can be caught by a stricter review and
| scrutiny.
|
| Papers that are properly done all the way through, but with
| faked data meant to push an agenda, can be disproven by counter
| research.
| fc417fc802 wrote:
| > Systematically well-done research with fake results or data
| is much rarer than just... lazy bad science.
|
| Genuine mistakes, logical errors, and other oversights are
| even more common than that. For all the issues it has, peer
| review is quite good at catching the things that it's
| intended to catch.
| yummypaint wrote:
| Have you personally reviewed a non-zero number of papers? What
| is this statement based on? For a thread ostensibly about
| science, the comments are disappointingly lacking in evidence
| and heavy on vibes.
|
| Maybe people could learn about what peer review is before
| posting their strong feelings about it? The purpose certainly
| isn't to replicate people's experiments, that happens after
| publication and not by referees. One of a reviewer's duties is
| to look at whether the study could be replicated given the
| included information. That is a very different thing.
|
| Also, just because something has made it past peer review also
| doesn't mean it isn't controversial in the field.
| jostmey wrote:
| I've only peer reviewed a few papers. I've submitted my share
| of papers for review
|
| This is my academic profile
| https://scholar.google.com/citations?user=bxn78bkAAAAJ&hl=en
| dr_dshiv wrote:
| We need more scientific societies. Modern peer review is super
| modern. Go back to the origins of science and it was all about a
| real community--setting high standards and having just a few
| people decide what to publish. It wasn't trying to be "objective"
| -- it was just high standards, determined invisibly by the
| members of the society. "Should we publish this?" asked the
| society.
|
| Alas, scale ruins everything. Nevertheless, I strongly believe
| "science is friendship"
| aDyslecticCrow wrote:
| That model had some major issues. Too many opposing theories to
| the "established norm" were dismissed, among things we now know
| were wrong all along.
|
| Although it could raise the standard of the process itself
| (methodology, writing) to very high levels, it restricted
| innovative ideas or unpopular outcomes.
| striking wrote:
| Sounds like a recipe for serious inequity to me.
| JumpCrisscross wrote:
| > _Sounds like a recipe for serious inequity to me_
|
| If it works better, it works better. The problem with the
| society method is it didn't work better than a decentralised
| scientific system.
| aspenmayer wrote:
| One of the weaknesses to such a system is human nature. We
| want to believe, which leads to farces like Piltdown man,
| which was a farce by a man called Charles Dawson (not
| Darwin, not the natural selection guy, though I did a
| double take at first), against the Geological Society and
| society at large. The farce wasn't definitively disproven
| until 41 years later, which is quite the downside risk of
| the gentlemen's club research group gatekeeping scholarly
| research, but the more things change, the more they stay
| the same. This is a good thing which seems like it's in
| response to grumblings about irreproducibility and bias,
| but those issues will likely always be with us and must be
| considered anew each time an experiment is designed, and
| each time a print run is cut.
|
| I'm having trouble finding a downside besides vote buying
| or voting rings, now that which way one has voted is now
| attributable. Can you think of any risks under the new
| system?
| HPsquared wrote:
| The trouble is, scientific research is now a livelihood and
| career for must published researchers. It's not an aristocratic
| hobby anymore. The incentives are all different.
| eig wrote:
| A good step forward. Reading the referee reports and rebuttals
| from papers that previously opted in to transparent reviewing was
| incredibly useful to my own paper publishing process. In many
| ways Nature is ahead of Science on this.
|
| I hope that making things transparent will help reduce the
| situation where big labs have an easier time getting their work
| into high impact journals through relationships with the editor.
| aDyslecticCrow wrote:
| This is amazing. The trust of peer review as a stamp of quality
| among academics is dwindling, and distrust of science among the
| population is growing (within the increasing politicisation of
| some areas of research). Transparent peer review raises the bar
| for both academics and enhances the potential trust in the
| process.
|
| This is desperately needed as AI could further deteriorate the
| quality of science if the publishing process is not made more
| strict. This represents a significant step forward in rigorous
| science. I hope other publications follow suit.
| thfuran wrote:
| >and enhances the potential trust in the process.
|
| I highly doubt it's a meaningful factor in public trust.
| Ar-Curunir wrote:
| The general public does not distrust science because peer
| reviews are not public.
| jfengel wrote:
| Of the people who distrust science, how many of them have ever
| read a scientific paper?
|
| I suspect the number is low. If that's the case, they're
| unlikely to be more convinced by the presence of published peer
| review, either.
| oerpli wrote:
| I would go further: Anyone who has published a "scientific
| paper" in the last decade or so either "distrusts science" or
| is more likely than not a mid-wit at best.
|
| Your posting doesn't give me the impression that you're very
| familiar with "science".
| iorrus wrote:
| Absolutely, the quickest way to lose faith in "The Science"
| is actually to do Science in a formal research
| institute....
| petschge wrote:
| Seeing how the sausage gets made makes you realize ALL
| the downsides and things you'd rather not have known.
| That doesn't mean you can think of a way to fix science.
| Let alone get the required funding to make an actual
| attempt.
| probably_wrong wrote:
| Isn't that true for everything, though? If I weren't a
| software developer I wouldn't know that I have to worry
| about questions like "has this plane been rebooted in the
| last 51 days?" [1] or "does this bank offer anything
| other than SMS as second factor?".
|
| Maybe structural engineers feel safer after their
| Master's when they traverse a bridge, but I bet that's
| more the exception than the rule.
|
| [1] https://www.theregister.com/2020/04/02/boeing_787_pow
| er_cycl...
| kergonath wrote:
| > Anyone who has published a "scientific paper" in the last
| decade or so either "distrusts science" or is more likely
| than not a mid-wit at best.
|
| That is very unhelpful, to say the least. The amount of
| noise has increased, but it does not mean that the
| scientists who know their subject disappeared. They are
| still around and not any less bright than their
| predecessors were 30 years ago.
| NoMoreNicksLeft wrote:
| >Of the people who distrust science,
|
| Why should anyone trust science? Skepticism should always be
| the default position. Putting it on a pedestal to be
| worshipped is what led us all into this mess. If science
| needs trust to work, then whatever it is doing is something
| I'd like to see fail.
| JumpCrisscross wrote:
| > _Why should anyone trust science?_
|
| When people talk about those who distrust science, they
| aren't referring to the carefully sceptical. They're
| talking about people who come to a conclusion and then
| reject any evidence against it.
| mikeyouse wrote:
| Right. RFK supporters aren't carefully weighing evidence
| to arrive at their conclusions, they are blindly
| following someone who is a mindless contrarian with a
| 30yr vendetta against vaccines. No amount or quality of
| peer review is going to change any of their minds.
| JumpCrisscross wrote:
| > _RFK supporters aren't carefully weighing evidence to
| arrive at their conclusions, they are blindly following
| someone_
|
| I'd actually be curious what fraction of his supporters
| share his views on vaccines. It strikes me as more likely
| that they like one of his random, idiosyncratic views and
| are willing to excuse the anti-vaccine nonsense to get
| those policy outcomes.
| fc417fc802 wrote:
| That, but also people who take the blanket view that none
| of the work produced by the scientific establishment is
| trustworthy. An ironic stance considering that it's the
| inverse of the blind worship that they accuse others of.
| kergonath wrote:
| > That, but also people who take the blanket view that
| none of the work produced by the scientific establishment
| is trustworthy.
|
| That's a red herring. No scientist actually says that.
| What we say is that on some subjects the evidence is
| overwhelming and to overcome the current understanding
| you need compelling evidence and theories, not screeches
| about bias and liberal elites.
| kergonath wrote:
| > Why should anyone trust science? Skepticism should always
| be the default position.
|
| Skepticism is not anti-scientific. Hell, distrusting
| results and theories is not anti-scientific. Distrusting
| the scientific method is. There is a difference.
|
| > Putting it on a pedestal to be worshipped is what led us
| all into this mess.
|
| Scientists did not ask for this. Amongst all high profile
| politicians, those who whine about science becoming
| political are those who made it so, by taking contrarian
| positions to rile up their voter base. Most people who make
| this point are not being honest. If you are, you should
| make specific arguments rather than rehash propaganda.
| aDyslecticCrow wrote:
| I think there are a few groups and reasons of distrust. Some
| more or less valid.
|
| Those that distrust authority as a whole and lean into
| conspiracies cannot be saved with this kind of thing.
|
| But i think news and science are having similar perception
| issues recently.
|
| Distrust for news growing among the average population (for
| some good reasons). People are loosing faith in the
| objectivity of established media organisations. Most people
| are exposed to science through these traditional news.
|
| So adding back some sense of confidence and authority to
| scientific institutions is very valuable to non-academics.
| Even if they themselves would not read the papers or revews.
| kergonath wrote:
| > The trust of peer review as a stamp of quality among
| academics is dwindling
|
| The thing is, peer review is not a stamp of quality, and never
| was. It is just the basic level of due diligence. The referees
| cannot reproduce the results most of the time for a lot of very
| good reasons. They are here as a sanity check, to ensure that
| the work avoided common pitfalls and actually make sense.
|
| What most people do not understand is that articles are not
| good because they are peer-reviewed; it's the lack of peer
| review that is a red flag. Amongst reviewed articles, a lot of
| them will turn out to be wrong or flawed in ways that are
| impossible for the reviewers to find out.
| probably_wrong wrote:
| I want to stick my neck out to say that this has the potential of
| being very bad for science.
|
| Imagine saying "no" to a researcher with a big social media
| profile. Imagine 4chan coming at you with style-detection and
| deanonymization tools simply because their favorite racist or
| antivaxer got their nonsense rejected and sent their followers
| after you. And this is not just me feeling this way - quoting
| myself from a previous comment, and according to the ACL's 2019
| survey [1], "female respondents were less likely to support
| public review than male respondents" and "support for public
| review inversely correlated with reviewing experience".
|
| A measure that women ~~and inexperienced researchers~~[2] do not
| support is a measure that favors only those who are already part
| of the club.
|
| [1] Original here (currently offline):
| http://acl2019pcblog.fileli.unipi.it/wp-content/uploads/2019...,
| summary here:
| https://www.aclweb.org/adminwiki/images/f/f5/ACL_Reviewing_S...
|
| [2] This part has been correctly pointed out as being wrong.
| JumpCrisscross wrote:
| > _Imagine saying "no" to a researcher with a big social media
| profile_
|
| "The identity of the reviewers will remain anonymous, unless
| they choose otherwise -- as happens now."
|
| (Also "support for public review [being] inversely correlated
| with reviewing experience" means inexperienced reviewers are
| _more_ likely to support it. Not less.)
| probably_wrong wrote:
| You are correct about the second point - I'll strike it
| through once I find out how.
|
| As for the anonymous part, that's why I wrote "with style-
| detection and deanonymization tools". If the Internet could
| find Shia Labeouf's flag in a day [1], could they find a
| reviewer based on their writing?
|
| [1] https://www.dailydot.com/unclick/4chan-shia-labeouf-
| secret-l...
| herewulf wrote:
| Translate into $RANDOM_LANG and then back to English. The
| perfect prose obfuscator.
| b59831 wrote:
| What is the alternative?
|
| Or are you just for creating classes of people that just can't
| be critiqued in any circumstance?
|
| This kind of sounds like, 'Wont anyone think of the grifters?!'
| probably_wrong wrote:
| I know every area is different but the "grifters" in the area
| of Computational Linguistics (the ACL) are "any volunteer[1]
| whose paper has been accepted at least once", meaning anyone
| from PhD students to professors and industry researchers.
|
| Not all academia is Elsevier.
|
| [1] This policy has been altered recently, though, and now
| submitting a paper comes with reviewing duties.
| leereeves wrote:
| > support for public review inversely correlated with reviewing
| experience
|
| "Inversely correlated" means that inexperienced researchers
| were supportive, not:
|
| > A measure that ... inexperienced researchers do not support
| ... a measure that favors only those who are already part of
| the club
| lenerdenator wrote:
| I only got as far as undergraduate research assistant in the
| academia racket, so maybe I'm not "with it" in the way serious
| researchers are, but is this to suggest that this wasn't being
| done, by default, on everything already?
___________________________________________________________________
(page generated 2025-06-16 23:00 UTC)