[HN Gopher] Double-anonymous peer review reduces reviewer bias, ...
___________________________________________________________________
Double-anonymous peer review reduces reviewer bias, finds three-
year trial
Author : rntn
Score : 77 points
Date : 2023-04-04 13:39 UTC (9 hours ago)
(HTM) web link (www.britishecologicalsociety.org)
(TXT) w3m dump (www.britishecologicalsociety.org)
| mike_hearn wrote:
| I scan read the paper.
|
| * It seems OK. Large sample size, fairly simple statistical
| treatment, clear and (mostly) honest writeup of their results.
|
| * About half of reviewers could correctly guess who the authors
| were despite the double blinding.
|
| * The motivation was the widespread belief in their field that
| women are discriminated against by peer review. In reality they
| found a slight bias in favor of women. (Their abstract and press
| release says they found no bias, but on p11 they say
| differently).
|
| * They also found no racism. Review results for people from non-
| western countries didn't change.
|
| * The difference is all due to a strong positive bias towards
| researchers from rich countries. This is probably "prestige
| bias", but can also just be that people go softer on people they
| know already because the reviewers are mostly western.
|
| * Optional double blinding will therefore be ineffective, it has
| to be mandatory.
| WFHRenaissance wrote:
| Okay, but now do it for PRs.
| entropicdrifter wrote:
| "This isn't in the code style of <Reviewer number 4>. Please
| redesign from first principles and rewrite your entire project
| to reflect the preferred coding style"
| Qem wrote:
| That should be the default for a long time already.
| bilbo0s wrote:
| My cynical view is that the reason it isn't is because so many
| papers would not survive the review process. And all the
| consequences of that, up to and including impacts on getting
| degrees and even tenure.
| Red_Leaves_Flyy wrote:
| These people do crap work so instead of holding them, and
| ourselves, accountable for our crappy work, we're going to
| lower our standards and hope the public doesn't notice? What
| happens when the public starts noticing the consequences of
| bad research having tangible impacts on the real world? Just
| look outside, the anti science rhetoric is loud among many
| groups.
| SubiculumCode wrote:
| meh. Experts in the field often know who is pursuing what lines
| of inquiry.
| m000 wrote:
| Experts may indeed "de-anonymize" a paper if they want. But:
| (a) not all of your assigned reviewers may be such experts, (b)
| experts often delegate some reviews to their PhDs/postdocs who
| are more susceptible to being influenced by author's
| name/affiliation.
| bilbo0s wrote:
| (b) is especially prevalent.
| Overtonwindow wrote:
| This is an excellent step in the right direction, but will it
| help reduce the absurd volume of studies and the "publish or
| perish" paradigm?
| xiphias2 wrote:
| By the time that problem is solved, GPT-8 will be out, and even
| GPT-4 can be used to help reduce the number of studies (making
| a table of the studies and asking questions about it).
|
| Reproducibility is getting better, but still a huge problem (we
| can't forget the billions of dollars of funding that went to
| Alzheimer for a scam paper)
| mike_hearn wrote:
| Is reproducibility getting better? The last semi-rigorous
| treatment of the topic I saw suggested there was almost no
| change over the past decade, at least in the social sciences.
| It did however use a proxy for reproducibility.
| xiphias2 wrote:
| Social sciences are inherently political: facebook, tiktok,
| tinder and onlyfans have all the resources and data to help
| people getting less depressed (for onlyfans it would be
| just shutting themselves down, as it's hard for me to think
| that there's any honest real human connection going on
| there, which makes both sides depressed over long term).
|
| Social workers are just trying to reverse some of the
| damage companies and governments are doing, but I don't
| think it will help, even I honestly think they are
| generally good people doing their best.
|
| Maybe disrupting the ad model that just started recently
| can help, but I believe social studies have no chance to
| fight against these things.
| anamexis wrote:
| I don't see why it would. It doesn't change the incentives.
| mike_hearn wrote:
| Fixing that would require a fundamental change in academic
| incentives. But if you fixed that, it wouldn't be academia
| anymore. After all, how else do you judge their output?
| Tangible results that solve a problem for non-researchers?
| That's what companies pay researchers for. To a large extent
| academia justifies its own existence by the incentives it gives
| to researchers.
| ghc wrote:
| > But if you fixed that, it wouldn't be academia anymore.
|
| That's a bold claim worthy of consideration. To assess it, we
| need to first ask: have the academic incentives always been
| "publish or perish"? And the answer is...apparently not,
| according to "Scientific Autonomy, Public Accountability, and
| the Rise of "Peer Review" in the Cold War United States".
| This paper details how the current paradigm is a product of
| political anti-science-funding sentiment in the 1970s.
|
| That tracks with anecdotes from Einstein, Higgs, and Fuller
| regarding distaste with the modern peer review system.
|
| Unfortunately peer review looks like it's actually a policy
| issue, because it's tied to government funding, so the change
| in incentives would have to start there. I don't have any
| answers but at the same time I think it's worth considering
| the fact that "publish or perish" is not the natural state of
| academia, but rather a recent phenomenon that appears to
| coincide with the reproducibility crisis and a decline in the
| number of large scientific breakthroughs.
| [deleted]
| xhkkffbf wrote:
| Some fields like Computer Science have had double blind review
| for some time. Others, not so much. Still, smart reviewers know
| who is working in which area. They can usually identify the other
| person pretty quickly. It's not always 100% accurate, but the
| people who you want doing the reviewing will be the ones who know
| the others in the field pretty well.
| floren wrote:
| Or a review will say "Why didn't you cite Foo, an important
| work in the field?" and then you look at the committee list on
| the conference site and see that Dr Johnson worked on Foo. It's
| happened to me, but what can you do.
| hedora wrote:
| I've been your proverbial Dr Johnson many times. There are at
| least even odds that some other reviewer told you to cite me.
| When appropriate, I try to find follow on work, or point to a
| paper I cited that more people should read.
|
| Also, I think people are overestimating how much reviewers
| care who the authors of the papers they are reviewing are, at
| least in my field.
|
| For one thing, the papers are often written by a student, so
| writing style isn't a great giveaway. Also, the advisor
| probably appreciates a thoughtful rejection of the crappy
| student paper. The alternative is getting it accepted,
| followed by a mad dash by the advisor to copy edit it and add
| experiments so the work isn't embarrassing. (Tenure clocks
| are the main counter example of this, but junior professors
| aren't exactly in a position to bully the reviewers!)
|
| People could try to block work that would scoop their ongoing
| paper, but I don't really see that happening in practice in
| my field. Double blind doesn't help with that, of course.
|
| However, it would pretty rapidly tank that reviewer's
| reputation, since the other reviewers would see the bad
| behavior, and they know each others' identities.
| quelltext wrote:
| I don't know. I can understand that as a reviewer, if you see
| work that relates closely to your own prior work, you would
| be inclined to point the authors to that work.
|
| They may simply have missed it or failed to realize the
| relationship before so blindly saying "it's fine I will just
| ignore that they skipped over my own work" when it is in fact
| related doesn't seem like the right approach.
|
| If it really is unrelated I doubt anyone feels the need to
| point to their own research.
| scythe wrote:
| When I worked for MDPI (please lower your pitchforks), we
| had a guideline, IIRC, that the reviewers of a paper were
| supposed to not be collaborators with each other, in
| addition to the usual requirement that they not be
| collaborators with the authors. ("Collaborators" meant
| coauthorship within five years ago). In principle, this
| means that if a citation was necessary and you were an
| author on it, you could hope that your co-reviewer(s) would
| recommend it.
|
| In practice, reviewers did sometimes request citations of
| their own works. As an editor you could reject the whole
| review report, but you were on deadlines and quotas and you
| typically had a hard enough time finding reviewers anyway.
| In my experience, most of the practices that researchers
| gripe about in MDPI are results of onerous working
| conditions rather than any kind of laxity in the culture;
| everyone was regularly encouraged to be serious about
| scientific integrity and seek out the best reviewers, and
| practically all of the people I worked with were quite
| intelligent and concerned about doing a good job, but it
| was also an unspoken expectation to work after hours,
| Saturdays, to always be sending an email, etc; exhausted
| people make mistakes.
| superb-owl wrote:
| A friend of mine was sexually assaulted by a man who would be a
| very likely reviewer for her papers. It basically forced her out
| of the field.
|
| This would be an important step forward.
| hedora wrote:
| That's horrible, but she could have just listed him as a
| conflict when submitting (at least in CS; I assume all fields
| allow authors to disclose reviewers with conflicts).
|
| In the worst case, a PC chair would occasionally ask about it,
| so it would follow him for years.
| saghm wrote:
| Given that this person already has shown they don't care
| about harming others, it seems reasonable to be worried about
| retaliation if they found out they were being listed as a
| conflict. I'm not sure about anyone else, but I don't think
| I'd personally be willing to bet my safety on the hope that
| putting someone down as a conflict wouldn't leak somehow; all
| it would take is one person to not realize the severity of
| the reason they were listed to get curious and ask their
| colleague in their field if they knew why they were listed as
| a "conflict".
| flangola7 wrote:
| That's horrible
___________________________________________________________________
(page generated 2023-04-04 23:02 UTC)