[HN Gopher] Strife at eLife: inside a journal's quest to upend s...
___________________________________________________________________
Strife at eLife: inside a journal's quest to upend science
publishing
Author : mfld
Score : 71 points
Date : 2023-03-18 09:54 UTC (13 hours ago)
(HTM) web link (www.nature.com)
(TXT) w3m dump (www.nature.com)
| LudwigNagasena wrote:
| That's a weird change. Publishing reviews alongside papers is
| good. Removing the "stamp of approval" from the whole process is
| weird. Imagine if every student who managed to enroll into a
| university would receive a diploma; and every course would only
| provide a review of a student instead of grades. That would just
| make _everything_ much harder.
|
| Yes, it may seem wrong to put things into black and white
| buckets; but many our decisions inevitably are binary and we have
| to make them no matter how hard it is.
| Vinnl wrote:
| They're not removing the stamp of approval:
|
| > with a short editorial assessment of the work's significance
| and rigour
|
| However, not getting the stamp of approval will no longer lead
| to the work not being accessible to people.
| LudwigNagasena wrote:
| But every paper can already be accessed by anyone if its
| authors decide to put it on arxiv, GitHub, their personal
| website or anywhere else they wish.
|
| So does that mean that the only practical difference is that
| now when you list your papers for a promotion or tenure
| review you will write "eLlfe (approved)" instead of "eLife"?
| Then what is all the fuss about?
| Vinnl wrote:
| Yes, but now the authors don't need to manually decide to
| also share their research elsewhere - something they've (in
| many, but not all, disciplines) been trained not to do as
| publishers traditionally wouldn't allow it.
|
| The fuss is about publication and approval being seen as
| linked in the academic hivemind. Even if you're able to see
| them as separate steps yourself, that doesn't mean that the
| academic world at large sees them as such, so if this fails
| to gain widespread acceptance, the reputation of eLife-the-
| journal might suffer, and having been published there will
| no longer look as good on your CV as it currently does.
| ttpphd wrote:
| In other words, the people who are afraid of reputational
| loss are the ones who have benefited the most from the
| broken system of academic publishing.
|
| I won't cry for them.
| mattkrause wrote:
| 1. There's still some hurdle before new-eLife will send
| your paper out for review. This is not necessarily
| "quality" or "impact" level; they say it will depend on
| whether they can provide a helpful and effective review.
|
| People had very mixed experiences with old-eLife's
| editorial decisions, and some felt it was a bit clubby:
| papers from some labs sailed through to acceptance while
| similar work from others got editorially rejected (without
| review) for seemingly minor reasons. Their policy against
| requesting new experiments could certainly be used as a
| cudgel: an editor can just say you're not convinced by a
| control and the paper's DOA.
|
| Thus, they are both uniquely positioned to make sweeping
| changes _and_ have people doubt them.
|
| 2. The reviews use a controlled vocabulary, so I think
| you'd say eLife (excellent approach, major impact) or
| whatever the words are.
| ttpphd wrote:
| The whole point is that journal prestige is harmful and fake.
| It's prestige by association. That's not how science should
| work.
| seydor wrote:
| The problem is that science has not figured out how to work
| without this system.
| atdrummond wrote:
| Sure it has. The modern "scientific publishing" industry is
| much younger than science.
| ttpphd wrote:
| Lolol
|
| Yeah we have. It's called reading the paper and applying
| judgement.
| tpoacher wrote:
| Maybe _you_ do that.
|
| But my HR department which made me write "publish X
| papers in top-tier journals (see list)" as a condition in
| my probation agreement most certainly does not.
|
| And alas, as long as I'm under probation, _their_ opinion
| carries far more weight in the debate than any moral
| argument you could come up with.
| ttpphd wrote:
| Yes I do that, thank you.
|
| Thanks for clearly stating how journal prestige is
| harmful to academics and scientists.
| seydor wrote:
| When was the last time you were hired like that?
| ttpphd wrote:
| About 8 months ago :)
| warkdarrior wrote:
| In some fields, there are thousands of submissions daily
| to Arxiv. Good luck staying up to date.
| ttpphd wrote:
| That's not a problem to be fixed through fake journal
| prestige.
| Turing_Machine wrote:
| > every course would only provide a review of a student instead
| of grades
|
| Would it? You'd think that a hiring manager would rather see a
| description of what the student accomplished, and with what
| level of skill, rather than a rather opaque letter grade.
|
| Of course, that would make it tough to run "cattle car" courses
| with thousands of students being graded with multiple-choice
| tests. I don't think that would be a bad thing, either.
| canjobear wrote:
| > Of course, that would make it tough to run "cattle car"
| courses with thousands of students being graded with
| multiple-choice tests.
|
| It will be easier with GPT providing the evaluation.
| ttpphd wrote:
| Ugh stop. Not every conversation has to be about GPT.
| markkitti wrote:
| The presence in a journal is not a grade. It is a "pass".
|
| Perhaps some journals only let grade "A" papers pass, but
| sometimes that same journal has an editor who might let their
| buddy's article slip through. If one is part of the old boys
| club, this is a nice situation.
|
| What eLife wants are signed grades from editors and reviewers.
| The editor is supposed to attach a brief summary in a few
| sentences summarizing the reviews. Hopefully one might bother
| to read a sentence or two to evaluate a paper instead of merely
| looking at the cover of the journal.
|
| eLife's move here is basically a statement that the system is
| corrupt. Those wanting to fight the corruption are trying to
| increase transparency and reduce arbitrary decisions.
| etrautmann wrote:
| The change just means that the stamp of approval is not the
| journal name, which is fine. If you think about it, that system
| is a bizarre and lazy way to attribute value to a piece of
| work. The book publishing model figures out which books are
| great without caring at all about who publishes something (for
| the most part)
| Vinnl wrote:
| eLife is a treasure. eLife isn't just another "reputable brand"
| in publishing; they're aiming for a better publishing model.
| Thus, fears like
|
| > They worried it would diminish the prestige of a brand they'd
| worked hard to build
|
| are disappointing to hear. eLife's good brand is a way to get
| their foot in the door, to change the system from within, but the
| dependence on good journal brands in the academic world is the
| primary reason that publishers are able to extort the academic
| world and lock the outcomes of publicly-funded research behind
| paywalls.
|
| It's hard to not become the thing you set out to fight once
| you're in the system, but I appreciate that eLife and Eisen are
| still pushing to change it from within, and I hope they succeed,
| and that they overcome pushback from the vested interests.
| Beldin wrote:
| This is an interesting development. The current publication
| process is quite broken, including due to the outsized importance
| given to the broken peer review system.
|
| Wrt that: the NeurIPS conference did a test once, splitting the
| PC in two and having both review the same batch of 100 papers
| (and arrive independently at a decision). Roughly speaking (top
| of my head), the two sub-PCs agreed on the top 15% and the bottom
| 25%. That is: both recognised the obvious good stuff and the
| clear rejects.
|
| For the remaining 60%, it basically was 50-50: one sub-PC would
| arrive at accept and the other at reject.
|
| Which, to me, points out 2 things: (1) there is something very
| broken for about 60% of submissions... (2) but the process works
| as we want for the other 40%.
|
| So I don't think accepting everything would be ideal, but I'm
| more than happy to try.
| timkam wrote:
| I'd say the experiment (relying on the summary provided here)
| shows what works: - Papers that are obviously poor are
| rejected. - Papers that are uncontroversially good are
| accepted.
|
| From a CS point of view, the problem is that in many
| communities, what is 'uncontroversially good' depends on
| subtleties only known to insiders who are willing to comply
| with the quirks of a community. And most communities are very
| defensive when it comes to shielding their top venues from
| relative outsiders, who are not sufficiently close to the inner
| circle of key players. As a consequence, getting the top-venue
| stamp (required to please the administration) becomes a social
| game. I think having a more open review process and stronger
| editorial control (vs. reviewer power) can mitigate this
| problem to some extent, for the following reasons: 1. When a
| paper is good, but upsets the subjective feelings of reviewers,
| the system should be implemented in a way that there is a high
| social cost for killing a paper because of subjective/cosmetic
| issues. Having venues where papers that are sent out to
| reviewers are most likely reasonable and should only be killed
| because of 'hard' flaws creates such a system. 2. When a paper
| is bad and the editors push it through, publishing details of
| the paper's editorial flow would impose social costs on the
| editors. (The reviewers, who may be potentially early in their
| career and may need protection, may even remain anonymous, the
| editors, who typically enjoy a strong standing in the
| community, should not.)
|
| In CS, journals tend to operate a bit closer to this model than
| conferences, which I think makes the journal submission and
| revision process more meaningful. To conclude, I don't think
| getting rid of rejects is the solution, but rather editorial
| policies that encourage reviewers to either expose fatal
| technical flaws in a paper or help the authors improve the
| presentation to achieve a better end-result.
| trinsic2 wrote:
| This sounds reasonable to me as a lay person.
|
| Humans have flaws that operate under the surface and it's
| good to have a system that minimizes the effect of ego on the
| dissemination of ideas.
| querez wrote:
| as a long-time NeurIPS reviewer: the problem is that most
| submissions are kind of "meh", and everyone knows it. They're
| usually well-done papers (in the sense that the math checks out
| and the empirical evaluation is is done well enough), but it's
| pretty obvious that the paper isn't going to be influential.
| Mostly, these are papers of the "this is a cute idea that'll
| give you 0.1-2% improvements on some benchmarks people usually
| look at" variety. So depending on how strongly a reviewer
| believes that this might actually become influential if someone
| REALLY, REALLY digs down on this idea or not, you'll give it a
| "barely passing" or a "barely rejecting" grade. That's what
| most of these 60% usually are.
|
| I don't think the problem here is actually the "broken peer
| review system". I think this is a fairly natural development:
| if your PhD advisor insists that you have a few "good papers"
| before they'll agree to let you graduate, and "publication
| venue" is used as a proxy for "is a good paper" (which is very
| reasonable, because it might take years before better proxies
| such as "citation count" give a good signal), then you'll
| submit your paper to top tier venues and hope for the best. I
| don't really know how to fix this: Establishing things like
| TMLR (Transactions of Machine Learning Research -- a rolling-
| release journal meant to have roughly the quality of NeurIPS)
| might be a good way forward. Students get their stamp of
| approval by publishing there and NeurIPS can significantly
| increase their acceptance threshold. But once you do that,
| you'll risk that TMLR doesn't count as "good enough venue"
| anymore....
| DonsDiscountGas wrote:
| > (1) there is something very broken for about 60% of
| submissions
|
| My conclusion is that there is an inherent subjectivity
| involved in judging paper "quality". Which shouldn't be that
| surprising; if there were objective criteria we probably
| wouldn't need humans to do any judging. I dunno if you want to
| call that a broken system or not, but if this outcome is
| unavoidable then trying to change it won't improve anything. (I
| suspect that 60% isn't some magic lower limit and improvement
| is possible, but I suspect the lower limit is 20-40% at
| minimum).
|
| Realistically, this means that a single acceptance/rejection
| shouldn't decide the fate of a persons whole career. And they
| generally don't, it will be an average of many.
| Vinnl wrote:
| > So I don't think accepting everything would be ideal, but I'm
| more than happy to try.
|
| An important caveat is that this experience isn't _accepting_
| everything; it 's _publishing_ everything (or at least,
| everything that satisfies the desk check and is sent out for
| peer review). The whole point is separating evaluation from
| publication; they 'll still distinguish works that they think
| are significant and rigorous from those they do not, but all
| works will be accessible. That's the mental shift they're
| trying to make happen.
|
| (Disclosure: I volunteer for a project with a similar goal,
| that has received funding from eLife in the past:
| https://plaudit.pub.)
| als0 wrote:
| Looking at https://plaudit.pub and trying to understand it.
| It seems like a technical solution trying to solve a social
| problem (which I dare say is impossible). There's too much
| politics in academia for researchers to objectively endorse
| work of other research groups.
| Vinnl wrote:
| Thanks for taking a look. I think the problem you signal is
| already present in peer review as well, unfortunately: the
| politics also influences reviewer's accept/reject
| recommendations. With Plaudit, at least, the endorsements
| are public, which we hope will mitigate phenomena like
| review rings.
| als0 wrote:
| Blinded peer review at least enforces some degree of
| impartiality and anonymity, though obviously challenging
| in practice.
| bluenose69 wrote:
| It seems unlikely that members of tenure/promotion and grant-
| review committees would have a high regard for candidates who
| publish in journals that accept papers that garnered negative
| reviews.
|
| We already have blogs, arxiv, and similar. The main addition here
| seems to be that editors write something in addition to peer
| reviews. Where will they find the time to write these meta-
| reviews, and to deal with the inevitable complaints that may
| follow? This is asking a lot, particularly when it will be in
| service of a journal unlikely to have a good reputation.
| dahart wrote:
| We already publish in journals that 'garner' negative reviews;
| we don't get to see the review scores. Editors already comment
| on papers, and we don't see their comments. Everyone knows that
| some mediocre papers are accepted and some excellent papers are
| sandbagged behind the scenes, for less than honorable reasons.
| Maybe publishing the reviews and editorial comments would shed
| some valuable light?
|
| Why wouldn't tenure and grant committee members take into
| account high review scores in a journal that has high status
| and maintains an excellent review system? As long as eLife can
| maintain their status, maybe this could be a model for a
| positive change in science publishing, without making it any
| harder for people to get tenure, promotions, or grants?
|
| Side note, it seems like this thinking that we need to have
| peer validation gate-keeping the publishing process is part of
| the reason for the reproducibility crisis; the reason nobody
| dares writing reproduction papers is because it's so hard to
| get them published. Could help if journals accepted these by
| default?
| Vinnl wrote:
| > Side note, it seems like this thinking that we need to have
| peer validation gate-keeping the publishing process is part
| of the reason for the reproducibility crisis; the reason
| nobody dares writing reproduction papers is because it's so
| hard to get them published. Could help if journals accepted
| these by default?
|
| I think that's also where their distinction between
| significance and rigour comes into play. Reproductions might
| not be "significant", but if they're rigorous, they can still
| get _that_ stamp of approval.
| Metacelsus wrote:
| I submitted a paper to eLife a month before they announced the
| change. (It was published last month.) I'm worried that 5 years
| from now, hiring committees will think of eLife as "not a real
| journal".
| f6v wrote:
| With something like Scientific Reports having first-decision time
| of half a year this is inevitable.
| seydor wrote:
| I think it's wrong to defend the current state. Even journals
| like Nature are game-able to a large degree. You can publish
| quite mediocre science in them if you socialize with the right
| editors at the right times. High impact journals are too much of
| a social club to be trustable, and social actions take too much
| energy from scientists.
|
| eLife is the only high profile journal that experiments with the
| process. Their experiments may be in vain, because they still
| need to come up with an alternative measure that funding agencies
| will use to distribute grants. You can't replace peer gatekeeping
| with nothing, there has to be something else. That said, the
| "publish then review" model is great, it is a straightforward
| mechanism that has worked on the internet for decades. We have AI
| that can weed out the obvious garbage , and the open review
| process can enhance papers greatly.
| f6v wrote:
| The irony is that big publishers create new open access
| journals with fast review time to jump on the bandwagon. They
| know it all sucks and they don't want to miss the opportunity
| to make money.
| robwwilliams wrote:
| The review process is noisy and subject to much social
| engineering, lobbying, editorial PR, and quid pro quo if also
| serving as reviewer.
|
| Reviews are inherently noisy because there are usually no more
| than three (over-taxed, unpaid) reviewers. But papers are now
| often highly multidisciplinary. Luck of the draw applies with
| force at n = 3 or even n = 10.
|
| Reviews DO improve quality of final product, especially in the
| better journals. But there are still huge gaps in reviewing due
| to the underlying assumptions about "good" science.
|
| For example, the majority if experimental studies using mice and
| rats use only a single genotype or strain of animal and are
| therefore essentially n = 1. My inclination is to reject a great
| majority of such studies without serious review because they are
| just fancy case reports.
|
| So if you lucked out and did NOT get me as a reviewer you may be
| good to go at Science, Nature, or Cell. If you did not luck out
| then you will have to fight with the editor-in-chief to overcome
| my fundamental dismissal of case reports in experimental biology.
|
| I support the eLife direction. We have squandered 20 years
| already and the hegemony gets stronger. Eisen is a bold
| publication experimentalist.
| low_tech_love wrote:
| There are definitely some wrong things with peer reviewing and
| scientific publishing, but this is hardly a solution. Sure it's
| one way to do it; eLife is probably going to turn into a kind of
| arXiv with reviews. That is, a repository, not a journal. Whether
| that is good or not for them I guess it's subjective, but it's
| certainly not _the_ solution.
|
| I think one interesting model is the one by PLoS (especially with
| One). As long as the methods are valid, the research is original,
| and the language is minimally appropriate, the paper gets
| published. While they strive to eliminate the kind of subjective
| bias you get from small tightly knit communities (where
| subjective/invisible criteria are enforced in an informal way),
| they at least try to clear out the obvious junk. As a
| consequence, they too have become a half-repository, but at least
| one with a certain entry barrier. Then, at some point during the
| year, they make a collection of highlights or special picks for
| the previous year, which kinda work like what a conference would
| do.
| Vinnl wrote:
| It'd be interesting if you'd give arguments for why you think
| it's not a solution.
|
| > As a consequence, they too have become a half-repository, but
| at least one with a certain entry barrier.
|
| eLife will still have an entry barrier, but passing that
| barrier doesn't give you access to publication, but to a
| positive rating in the assessment of the significance and
| rigour. In other words, it will still perform the function of
| highlighting potentially relevant research, but it doesn't
| block non-highlighted research from being accessible.
| ttpphd wrote:
| The same people who made PLoS are the ones reforming eLife.
| Exact same.
| jhbadger wrote:
| Well, it's Michael Eisen, yes, and he was a co-founder of
| PLoS, but it's not the "exact same people" other than that.
| mattkrause wrote:
| I think that's _only_ PLoS One.
|
| My experiences with PLoS Biology and PLoS Computational Biology
| have been great, but very similar to other journals.
| jhbadger wrote:
| Yes. The idea of PLoS One is that people can subject directly
| to it, but also it serves as a second chance for papers
| submitted to the higher tier PLoS journals -- an editor can
| reject the paper from those while still suggesting that the
| authors submit it to PLoS One (if the reason for rejection
| was that it wasn't that exciting but still solid research).
| dahart wrote:
| Publishing reviews along with a paper sounds amazing to me,
| that's a part you never get to see in journals when you're not a
| participating reviewer. I'd love to have some reference points of
| expert opinion and critique come with each paper.
|
| I can understand the fears about submission quality. People do
| prefer submitting to 'elite' journals with low acceptance rates,
| because getting accepted becomes a validation. I wonder if
| publishing the reviews might take care of that anyway, if the
| audience will tend to read only the papers with high reviews.
|
| Maybe they could have the best of both by publishing a curated
| set of the top 10% of papers by review scores in a best-of
| version of the journal, and still publish everything in the main
| journal. The move to consider publishing everything makes sense
| in this new world where all papers are on Arxiv, and they
| frequently make rounds in the press and social media before
| getting reviewed. I would welcome the eLife changes if it helps
| quell the trend of completely unreviewed papers vying for
| attention.
| killjoywashere wrote:
| I review articles for an NPG journal. I have seen articles
| submitted that are essentially advertisements from a company.
| They don't bother to read the journal's guidance to authors or
| anything else. I don't see how promising to publish a review
| alongside that is a good idea. They're just looking for free
| press and you're rewarding their bad behavior. No one will see
| the review. They'll see the glossy reprint the company hands out
| at their booth at the conference, with your journal's emblem on
| it. I'm not against companies submitting articles, far from it,
| actually. But how am I going to confidently review a single
| spaced PDF? I don't even have room to jot notes. I am against a
| complete lack of respect for the process.
|
| In a case like that, my advice to the editor is not a review, it
| is to simply reject for failing to follow guidance to authors.
| Now, it may be that other journals have a better desk review
| process before sending out to reviewers, but I don't know.
___________________________________________________________________
(page generated 2023-03-18 23:01 UTC)