[HN Gopher] A Possible Fix for Scientific Publishing
___________________________________________________________________
A Possible Fix for Scientific Publishing
Author : dbingham
Score : 57 points
Date : 2022-08-07 12:58 UTC (10 hours ago)
(HTM) web link (blog.peer-review.io)
(TXT) w3m dump (blog.peer-review.io)
| t_mann wrote:
| Awesome idea! Only half-joking: something like the HN ranking
| algorithm could be great for that, maybe you can reach out to
| them if they can help (you might even try applying for YC).
|
| Some feedback: I think the division by disciplines needs to be
| stronger than tags - researchers are so specialized that most
| research even within the same discipline is often unintelligible
| for those specialized in a different sub-area. They need to know
| _roughly_ what to expect when they go to the platform, you don 't
| want to show physics papers to sociologists. Think of the
| preprint servers, they're also constrained to some disciplines
| (arXiv for mathematical disciplines, bioRxiv and medRxiv, SSRN
| for economics/social sciences...).
|
| In the bigger picture, I think moving towards such a model would
| be a good thing also for scientific debate in the public eyes.
| Going from the binary 'It's published' to 'It seems most experts
| in this area tend to agree with this paper's conclusions, with
| some important caveats' would give a much better picture of the
| state of our knowledge at the cutting edge, and perhaps also
| address the bias against negative results better than journals
| who pledged to do so can.
| jpmattia wrote:
| As an observation, I think "If you're not paying for it, you're
| the product" applies here.
|
| In traditional publishing, the people writing the papers are the
| product. Journals compete to get the best papers, and those that
| got the best papers in the immediate past tend to get the best
| paper submissions in the future.
|
| In open-access publishing, the readers are the product. Having
| good papers is only one way to build your product base, On the
| other side, there are a number of ways to market your product
| (pay-to-play, for example.)
|
| While I like open-access in principle, I also like the filtering
| from the traditional pubs to help me narrow a reading list.
| dbingham wrote:
| How would you apply that observation to Wikipedia? That's the
| business model we're aiming to take - crowdsourced,
| crowdfunded, non-profit, and community governed.
|
| And the reason for that is precisely because of the mis-aligned
| incentives in both the pay to access and pay to publish models.
|
| The filtering here would be accomplished by the tagging system
| and the voting system.
| CJefferson wrote:
| The problem is wikipedia is explictly rejecting what you want
| to accept -- original research.
|
| It takes minutes to check someone's reference says what that
| it claims they say. It takes days to check if a new piece of
| independant research is correct.
| jpmattia wrote:
| > _How would you apply that observation to Wikipedia?_
|
| I would notice that the Wikipedia reader base is a multiple
| of peer-reviewed scientific-paper base. While I don't know
| what the multiple is, I would guess the range is 1-2 orders
| of magnitude.
|
| Wikipedia claims its average donation is $15 [1], so a rough
| guess will imply $100-$1000 average donation for a sci pub
| model.
|
| That's on the income side. On the spending side: I would
| guess the major difference in costs would be the scientific
| editors, which exist in the traditional science publication
| model.
|
| There are probably large initial marketing costs associated
| with
|
| 1. finding the right target authors and convincing them to
| send a valuable contribution to the new site,
|
| 2. finding serious editors to manage peer review and separate
| the wheat from the chaff. (edit: I assume this is still part
| of the process. I think about something like Einstein's
| special relativity paper, and I'd have to guess it would have
| been widely downvoted as an internet crank trying to publish
| crazy ideas -- he did not have a reputation yet.)
|
| I see this as potentially expensive: The level of the first
| papers will determine how likely it is to succeed. Said
| differently: If the new site starts by publishing lame work,
| it's not like someone is going to want to publish there
| afterward.
|
| In any case: I think it is a tough and worthwhile problem to
| solve, and I wish you good luck.
|
| [1] https://donate.wikimedia.org/wiki/FAQ
| tgv wrote:
| I don't get the positive reactions. This sounds like: let's do to
| scientific publishing what social media have done to society. And
| that's make it worse. For fistful of dollars.
|
| - Academics advance based on your publication score. They will
| avoid your platform if it doesn't help them get grants or tenure.
| If they've got it, they will avoid it even more.
|
| - Only a few people will put time in reviewing and handing out
| points, where unfortunately reviewing means: three seconds of
| scanning and gut feeling. You won't be able to keep even the most
| blatantly fraudulent papers out.
|
| - There will be a few good articles that score well, and you'll
| defend your platform by pointing that out, but below that there
| will be a mass of highly rated papers that don't deserve it.
| They'll be there because a group with an agenda put them there,
| or because groups actively keep papers of "competitors" down.
|
| - As said by others: the "average citizen" isn't part of this.
| Never has been, and doesn't need to. Open Access is for fellow
| scientists and very specialized citizens.
|
| - "With over 10,000 academic journals, it's impossible for the
| lay public to track which journals are reputable and which are
| not." That's not true. Most researchers only look in a small
| number of journals and can distinguish wheat from chaff. But you
| won't have 10k journals: you're aiming at millions of reviewers.
|
| - Why use voting by god-knows-whom if you can use the same
| "trust" mechanism that's used today: citations? Because you'll
| only become a record keeper. There won't be a need to fund your
| system.
| dbingham wrote:
| Hey HN, I've had this idea for an open scientific web platform
| for years. It seemed like such an obvious thing to try. I kept
| waiting for someone else to build it, but to my knowledge no one
| else has. I finally saved up enough to take some time away from
| work and build a proof of concept. I'm looking for feedback on
| the PoC from scientists, scholars, researchers and academics to
| try to figure out if there is merit to this idea. I would deeply
| appreciate any and all feedback, and help getting this request in
| front of more of the target audience.
| eesmith wrote:
| Looks similar to the failed project
| https://en.wikipedia.org/wiki/Peerage_of_Science .
|
| > Here other scholars who have enough reputation in any of the
| fields you tagged the paper with can see the draft and offer
| feedback on it.
|
| Speaking for myself, I don't trawl through preprint servers
| looking for papers to review.
|
| As it is, editors have a hard time getting people to review,
| and even after agreeing to review, to get the review back in
| reasonable time.
|
| > This system gives scholars an enormous amount of control over
| who they solicit feedback from.
|
| How does it handle peer review rings, where authors collude
| with reviewers to get bogus positive reviews? This is already
| an issue now.
|
| > It gives reviewers an incentive to give solid, constructive
| review feedback - and rewards good reviewers for their efforts
| with recognition of their contributions.
|
| It offers reviewers an incentive to write a review that the
| paper author likes. I expect if reviewer writes "this paper
| duplicates work by Penzias & Wilson (1965) and offers nothing
| new - reject" then the author will reject it. While "Fix the
| typo on page 4, otherwise, go ahead and publish" will be
| accepted - even if the paper really is a waste of time.
|
| It's a lot easier to get surface-level proofreading responses
| than substantive domain-specific responses.
|
| > Down voters will be strongly encouraged to post a response
| explaining their downvote.
|
| Why aren't up voters strongly encouraged to post a response
| explaining their upvote? Why should upvotes not be equally
| visible?
|
| What's the negative consequence should someone not provide a
| response?
|
| > It is not a pre-print server. It is an attempt to replace the
| journal system with something open to its core, scholar lead,
| and collectively managed by the scholar community.
|
| Any reason to not develop this as an overlay journal on top of
| existing pre-print servers? Thus reducing a lot of the costs?
|
| Why would anyone use this system given PubPeer, and PeerJ, and
| eLife, and overlay journals, and Publons, and all of the other
| attempts at breaking the current system, why would someone
| choose this one?
| dbingham wrote:
| Hmm... well Peerage of Science was an overlay on top of the
| journals. It was basically a system the journals would pay
| for to mediate their own review. Maybe similar tech-wise, but
| very different goals.
|
| > Speaking for myself, I don't trawl through preprint servers
| looking for papers to review.
|
| > As it is, editors have a hard time getting people to
| review, and even after agreeing to review, to get the review
| back in reasonable time.
|
| Could we change that culture? The hope would be that we could
| build a culture of people helping each other review, for the
| simple reason that they want that help themselves when it
| comes time to for them to publish. With the reputation being
| more of a bonus.
|
| Keep in mind, review here is purely editorial feedback. It is
| not refereeing - that happens post publish through the voting
| system.
|
| > How does it handle peer review rings, where authors collude
| with reviewers to get bogus positive reviews? This is already
| an issue now.
|
| See above - review is not refereeing. It could be an issue in
| refereeing in the votes for sure though. The hope is that the
| overall weight of the community would override those rings.
| And in the long term we might even be able to identify them
| through analysis of the database. If we see the same group of
| people always up voting each other's down voted papers, we
| can flag that and take corrective action.
|
| > It offers reviewers an incentive to write a review that the
| paper author likes. I expect if reviewer writes "this paper
| duplicates work by Penzias & Wilson (1965) and offers nothing
| new - reject" then the author will reject it. While "Fix the
| typo on page 4, otherwise, go ahead and publish" will be
| accepted - even if the paper really is a waste of time.
|
| > It's a lot easier to get surface-level proofreading
| responses than substantive domain-specific responses.
|
| That may turn out to be true, but the thought is that the
| post-publish refereeing system will help incentivize people
| to take critical feedback seriously at the review step. After
| all, your goal is to publish something that will be well
| received. If you get good constructive critical feedback and
| ignore it, you risk being downvoted to oblivion and losing
| reputation.
|
| > Why aren't up voters strongly encouraged to post a response
| explaining their upvote? Why should upvotes not be equally
| visible?
|
| > What's the negative consequence should someone not provide
| a response?
|
| It's been asked before and it's definitely on the table. It's
| something I'd love more feedback on - should responses be
| required for all votes? Just down votes? Or (as I'm
| defaulting to) should we follow the StackExchange model of
| just encouraging responses on down votes?
|
| Seems maybe you would argue in favor of requiring responses
| on all votes? What's the reasoning?
|
| > Any reason to not develop this as an overlay journal on top
| of existing pre-print servers? Thus reducing a lot of the
| costs?
|
| We don't want it to be an overlay, because we want to
| ultimately replace the system. But it could make a lot of
| sense to work with the existing pre-print servers. I see no
| reason why we couldn't start as a pre-print and gradually
| evolve towards being the final publish, as long as we're
| clear about the end goal so that we don't just become "yet
| another pre-print".
|
| > Why would anyone use this system given PubPeer, and PeerJ,
| and eLife, and overlay journals, and Publons, and all of the
| other attempts at breaking the current system, why would
| someone choose this one?
|
| - PubPeer is just a comment overlay, can't read the papers
| (as I understand it).
|
| - PeerJ is still using the manual model, hence overhead and
| fees to publish ($1200) rather than diamond open access. This
| creates equity in access to publishing issues ( a lot of
| people cannot afford those fees ).
|
| - Elife, similarly, charged $3000 per article. These fees
| create massive equity in access to publishing issues.
|
| - Publons I wasn't previously familiar with, but it seems to
| be about just highlighting review work done for the journals,
| and not anything to do with open access.
|
| None of the attempts to break the current system (that I've
| found) seem to really be able to replace the journals by
| doing (almost) everything the journals do. At least, not with
| out just being a journal and facing the overhead problems.
| The one thing this project doesn't do that the journals do is
| curate reviews/refereeing, but the idea is that crowdsourcing
| could actually work better in the long run - though it's
| definitely going to bring its own set of problems that will
| need to be dealt with.
|
| But thanks! I hadn't heard of Peerage of Science, and Publons
| sounds vaguely familiar but I wasn't really familiar with it.
| I'm constantly learning more about what other efforts have
| come before and currently exist (there are a lot of them).
| And thinking through all the questions and counter points is
| always very helpful!
| CJefferson wrote:
| > Could we change that culture?
|
| The problem is doing a high quality review of a 30 page
| maths paper takes several days. In your system an author
| could then just reply with "nope", and my review
| disappears. Alternatively, my "downvote" after I spent two
| days finding mistakes can be cancelled out by one upvote
| from their office-mate.
| counters wrote:
| > Could we change that culture?
|
| In short? No. You probably can't.
|
| I hate being so blunt because I love the ideas you're
| proposing and have advocate for many similar things when I
| was still pursuing an academic career and now from outside
| of it (albeit in a scientific community where there is a
| very blurred line except for institutional affiliations).
|
| But you're not addressing the fundamental problem: there's
| such an over-saturation of PhDs and so few tenure-track
| positions in so many fields that there is one and only one
| objective for an early-career researcher: publish, publish,
| publish. That's how you build the portfolio to get called
| back for campus interviews for professorships; that's how
| you build your network and slide into funded grants before
| building your own, with connections to the program office
| or review committee that will help you get said grant
| funded. Although there are many worthwhile activities for
| an early career researcher to engage in (and, thankfully, a
| slowly growing recognition of the value of these
| activities, such as open source development and data
| curation), only a small subset of them feed into the
| ultimate goal: a tenured professorship.
|
| The critical shortage of reviewers these days, at least in
| my field, is partly due to social dynamics since the
| pandemic, but also very critically due to the perceived
| lack of value that serving as a review provides while
| you're in the heat of the rat race. It doesn't sound like
| your a practicing research scientist (meaning: you work in
| a job where producing research and publishing on this
| content is a primary metric against which your success is
| measured), but you should really talk to some across a
| diversity of fields. It takes a _lot_ of effort to provide
| a useful review. Effort that instead could be funneled into
| ones own research, for greater impact on your career. And
| if you're the author of a good paper, you don't need the
| other reviewers to "help" you with the process. It can be
| tedious and annoying sometimes, but hell - what difference
| dos it make if _one_ of your papers is getting held up,
| you've got 6 others ready to go by the end of the year,
| right?
|
| The problem is that it's not a "culture" issue. It's the
| fundamental practice of the modern scientific enterprise.
| The incentive structures are highly warped - a problem that
| many of us run into time and time again when we try to push
| through open science and reproducible science initiatives
| (and I'd definitely classify your effort in that group).
| When the fundamental practice and institutions change, the
| culture will change.
|
| If you'd be willing to bend and constrain your idea to more
| of a "Reddit of peer-reviewed literature" and focus on
| expert intersectional communities like ScienceBlogs, you
| might have something really novel here. There's a ton of
| value in the open-sourced curation mechanism. I've been
| involved heavily with /r/science in the past, and while the
| ecosystem of sub-reddits surrounding it definitely do a
| good job of "curating" science, it's not really the same
| thing as what you're proposing. It's a definite niche would
| could have a lot of value.
|
| But don't expect a lot of scientists to buy-in when you're
| not offering them anything in return. And again, to make
| this very, very clear: if your project doesn't help an
| academic get tenure, I don't see why they'd participate.
| eesmith wrote:
| > Could we change that culture?
|
| I don't see how this system can do that. I stopped caring
| about Stack Overflow up- and down-votes long ago, I've
| grown tired of reading stories about the background
| politics and voting for Wikipedia, I can easily see how the
| proposed system can be gamed, and I don't see how it can
| defend against systemic gender, racial, or other social
| biases.
|
| > for the simple reason that they want that help themselves
| when it comes time to for them to publish
|
| Which leads to collusion. "You scratch my back, I'll
| scratch yours."
|
| My reviews are anonymous. I want to make it harder to
| retaliate against me, and make it harder to use what
| authority I've gained to influence things, and I trust the
| editors to mediate.
|
| > review is not refereeing
|
| Shrug. Okay. The question is, why do people want to get
| points? If it's good to get points, then there will be
| collusion to get those points.
|
| > the post-publish refereeing system will help incentivize
| people to take critical feedback seriously
|
| I have not found that system to be all that effective.
|
| > Seems maybe you would argue in favor of requiring
| responses on all votes?
|
| That is not my argument. I don't know one way or the other.
| I observed that "transparency" is described as being
| important for negative votes, but with no reason given why
| it's not also important for positive ones.
|
| > We don't want it to be an overlay, because we want to
| ultimately replace the system.
|
| The advocacy objects to a system with "high fees",
| "paywalls", "pay-to-play", etc.
|
| Preprint servers don't have fees, paywall, or pay-to-play.
|
| Thus, there is no reason to suspect you want to replace
| preprint servers, nor reason for why it's important to do
| so.
|
| > None of the attempts to break the current system
|
| How much do you know about eLife and Michael Eisen?
|
| How hard is it to get a waiver for the $3000 fee? "To
| ensure that eLife's publication fee is not a barrier to
| publication we therefore offer a simple way for authors to
| apply for a fee waiver."
| https://reviewer.elifesciences.org/author-
| guide/fees#elife-p...
|
| It's also hard understand what you mean by "the current
| system", since I didn't realize pre-print servers were part
| of it. eLife requires preprint publication to bioRxiv or
| medRxiv.
|
| > there are a lot of them
|
| My question isn't "how does someone well-informed in all
| the different options choose yours?" but rather "given that
| most people aren't well-informed about the different
| options, why would they think to publish using your
| project?"
| dougmccune wrote:
| You might want to dig into what ResearchHub is up to
| https://www.researchhub.com/
|
| If you ignore the web3/crypto aspect, they've got a decent
| start on what seems like a reasonably similar platform. They do
| all their dev and community building out in the open, so you
| can join their Discord and listen in on their community calls,
| etc. Might at least give you some ideas on what to target or
| avoid.
| ISL wrote:
| Consider finding an academic niche in which to really make this
| product sing; a community that is receptive to the idea. If the
| PoC can be shown to work in one community, neighboring fields
| can then point at it and say, "see, this thing is working over
| in community X. Journal Y is going to take three months to get
| this reviewed, and this paper is kind of a throwaway -- let's
| try X and see how it goes."
|
| arXiv started in a narrow niche that absolutely needed it,
| perhaps the same will happen here.
|
| Long-term, I'm not sure how this system will pay for itself.
| Keep costs low and dig in for the long haul. Academia can have
| very low switching costs (a professor can try your service on a
| whim), but has enough inertia that it can take decades to turn
| the ship (deans need to know a prospective hire's impact factor
| in top-tier journals).
|
| Also, as a referee -- pay referees what their time is worth.
| For real. You'll get real good referees if you do.
| dbingham wrote:
| > Consider finding an academic niche in which to really make
| this product sing; a community that is receptive to the idea.
| If the PoC can be shown to work in one community, neighboring
| fields can then point at it and say, "see, this thing is
| working over in community X. Journal Y is going to take three
| months to get this reviewed, and this paper is kind of a
| throwaway -- let's try X and see how it goes."
|
| > arXiv started in a narrow niche that absolutely needed it,
| perhaps the same will happen here.
|
| That's very similar to my thinking - I'm actually hoping to
| start with the file drawer problem. When we're ready for open
| beta, put out a call for file drawered papers. It's also been
| suggested that we start as pre-print, while being clear about
| the ultimate intentions to become a journal, and go from
| there. Maybe even reach out to the existing pre-print servers
| to partner with them in some way.
|
| > Long-term, I'm not sure how this system will pay for
| itself. Keep costs low and dig in for the long haul. Academia
| can have very low switching costs (a professor can try your
| service on a whim), but has enough inertia that it can take
| decades to turn the ship (deans need to know a prospective
| hire's impact factor in top-tier journals).
|
| Noted. The thought is that it could be funded through
| donations, grants, and eventually by the institutions if it
| gains enough traction. But yeah, it's becoming clear that
| grants at the beginning may be the only way. We'll see how
| much we get in terms of donations once we hit open beta.
|
| > Also, as a referee -- pay referees what their time is
| worth. For real. You'll get real good referees if you do.
|
| ...well... the idea is actually to crowd source refereeing
| and review. And for people to do it because, well, it needs
| done. Everyone wants good reviews and everyone knows
| refereeing needs done, so we ask everyone to share the job
| equally and try to make it as easy and low friction as
| possible.
|
| The pre-publish review system is purely editorial review,
| aimed at helping authors prepare for publishing. The
| refereeing system is post-publish, it happens through the
| voting.
| Test0129 wrote:
| The other side of it is reputation. Academic publishing to a
| questionable journal (even if it's a good journal) can be a
| black mark on your career. You not only have to get over the
| hump of at least break even profit but also convince tenured
| professors stuck in their ways that your journal will gain
| and keep a positive reputation.
|
| This task is insurmountable for the most part. At least with
| the current way academia works. Current journals have been
| around for decades, sometimes even reaching the 100 year
| mark. Back in the ye olde days it was easy for anyone to
| publish anything (even non-academics). These days these
| journals are so deeply entrenched in the minds of academics
| it would be like convincing 10% of fortune 500 companies to
| switch from excel to your new cool software. Even it's it
| better they won't because of this mental block.
| dbingham wrote:
| Yeah, that may be. But it seems worth trying given all the
| negative impacts of the status quo.
| blep_ wrote:
| Could the solution be "don't actually tie it to the
| journal"? Have the journal treat itself more as an
| implementation detail than as an end in itself; you didn't
| publish a paper in the Journal of Lasers, you just
| published a paper about lasers.
|
| This is still a mental shift for academics to get over, but
| I wonder if it's a more feasible one.
| sseagull wrote:
| A few comments from a practicing scientist
|
| 1.) You have a lot of nice tech, but think long and hard about
| the social dynamics and how you can get from A->B. By far, the
| tech will be the easy part. Convincing established scientists and
| up-and-coming scientists who need publications in quality
| journals is much, much harder. As another comment says, try
| finding a niche first and build out. Particularly some less
| established fields.
|
| 2.) You mention hiring engineers, designers, devops, QE. This
| points to solving a social problem with technology. Consider
| marketing, or even social scientists who can guide you in
| understanding the communities you are attempting to influence.
| Become embedded in those communities, build trust, attend (and
| maybe sponsor) conferences, and then branch out.
|
| 3.) Sort of a pet peeve of mine (among many): You talk about
| making the studies accessible to the lay public (in terms of
| understandability). I feel this is a bad trend. Scientists need a
| way to talk to each other in their own language, without needing
| to lay out all the context every time. Academic publishing is
| that avenue. I have published many articles that will not be
| understandable to anyone without at least 2 years of graduate
| school in the field. They aren't groundbreaking, but can still be
| important. Science is very incremental in many areas. Although I
| 100% agree that it should be open and free.
|
| For the lay public, there are always books, blogs, and pop-sci
| news that put it into context.
|
| Anyway, I wish you success (truly, we need improvement over the
| status quo). It's easy for people to criticize, but think of all
| the things we don't criticize (which means we approve of them!)
| t_mann wrote:
| On the career / publishing perspective: I think you're
| overlooking the fact that an upload to such a peer-review
| platform would usually be considered non-archival, so the paper
| could still be published in a prestigious journal. It's
| basically a pre-print server with a comment function (which
| already exists in some sort, btw, researchgate.net), and most
| journals have accepted that they can't stop people from making
| their papers available through such channels.
|
| The bigger threat to this model is probably places like
| arXiv/SSRN adding similar functionality, but since this seems
| to be a non-profit project, it would probably mean mission
| accomplished.
| emailtho wrote:
| gonzo41 wrote:
| On your point 3. I don't disagree, I just observe that most
| disciplines of science use their specific journal writing
| styles to force a style of writing which is generally more
| about conforming to the norms of the discipline rather than any
| other benefit.
|
| Stuffy science can't answer this: Why would a youtube video
| with a DOI not be an acceptable form of publishing. (ignore the
| archiving issue for the discussion).
|
| You don't have to tone down content complexity, you only see
| that in sci-coms on social media because they are focusing on
| engagement rather than just using the platforms for their
| network effect.
| sseagull wrote:
| > Why would a youtube video with a DOI not be an acceptable
| form of publishing
|
| I can, at least personally. For some stuff video might work,
| but text has a much higher level of precision and information
| density than video. When writing an article, I will often
| edit sections many times until it has the right amount of
| information, references, and flow. That is much harder in
| video, and scientists don't know much about video editing as
| is...
|
| Also, when learning something, it typically takes multiple
| articles. I regularly have 5 articles open on my desk (yes,
| printed out), where I'm cross referencing equations, methods,
| etc, with one another. That is much harder to do on a
| computer, let alone with video.
|
| Again, maybe for some fields it makes sense (maybe?). But
| it's hard to beat ease of use of text.
|
| (Also, there's nothing wrong with conforming to a particular
| style. It's about consistency across authors, which helps the
| reader. Software projects have style guides, don't they? As
| well as traditional media outlets. Why shouldn't scientific
| publications?)
| idrios wrote:
| > Also, when learning something, it typically takes
| multiple articles
|
| This is where I think video would be most helpful,
| actually. To relate it to software, when I'm learning a new
| language or a new tool, a video helps me navigate and learn
| my unknowns. But when I'm already an expert on a topic,
| video is intolerable because it takes too long to get to
| the point and often doesn't have the information I'm
| looking for.
|
| Likewise for science, a video recording for materials and
| methods would probably be incredibly helpful for
| replication purposes. Also review papers might lend
| themselves better to video format because their purpose is
| to summarize existing research in a way that's digestible
| to someone relatively new to the field. It doesn't fit all
| purposes but it would make the science more accessible
| qwezxcrty wrote:
| About videos: I would argue it have a overlapping niche
| position with (either in-person or virtual) conferences. In
| optics community, SPIE and OSA do publish recordings of
| some (mainly hybrid) conferences. Those videos are indeed
| useful for learning about quasi-real-time works in progress
| from fellow colleagues (in slightly more depth than the PDF
| abstract) and save you from traveling.
| jinto36 wrote:
| There's already a peer-reviewed video-oriented journal: the
| Journal of Visualized Experiments http://jove.com/ The
| downside is of course it's not open-access by default, and it
| can be expensive even if you don't pay for open-access since
| they'll send a freelance videographer if you don't have
| equipment/time/confidence to make the video yourself.
| dbingham wrote:
| Thanks for this feedback - it's very helpful!
|
| 1) Yep. This I know. The tech is the easy part, by far. Any
| suggestions on which fields might be most open to this
| approach, and where to approach them?
|
| 2) Good point. At the moment, there is a lot of technical work
| to do, so that's where my head is. But if we get big enough to
| where we can afford it, community moderators who can help guide
| the community towards a collaborative approach and gently nudge
| back some of the egos I've already been told (many times over)
| are a problem were already on the list. Social scientists are a
| great suggestion that I had not thought of.
|
| 3) This may be an area where we disagree - at least partially.
| I understand the need to talk to peers in a mutual language,
| and I'm not arguing against that. My thought is that this
| platform could help provide space for translators. Papers and
| responses would be in the technical language of the field, but
| I've thought about including a (heavily moderated) peanut
| gallery below for translators to help the lay public understand
| the work.
|
| One of the issues that has been front and center in the
| pandemic is that a lot of bad vaccine science was easily
| accessible to the public, and when it was done intentionally by
| snake oil salesmen, intentionally made easily understandable.
| But the good stuff was locked up behind paywalls or hard to
| understand. There were translators on social media who did
| their best to answer the concerns but there weren't enough of
| them. There are a lot of people not currently in academia who
| have enough of a foundation to tell snake oil from the real
| deal and who could help translate, with access. Right now, they
| don't have access. So you wind up with PDFs of "1000 papers
| showing why vaccines are harmful" getting shared around and no
| easy way for people adjacent to the communities where they are
| being shared to counter them.
|
| Additionally, science is supposed to be an open process - we're
| supposed to trust it not because of appeals to authority, but
| because, in theory, we can replicate the reasoning and
| experiments that got us to our current understanding. Now, of
| course, replicating the experiments is mostly impossible for
| the lay person at this point, but someone who's interested and
| has the time should be able to follow along. We can't say, in
| the same breath, "You should trust science, because it's open
| and you can replicate it" and "Oh, you can't understand this."
| We need to create space for scientists to talk to each other,
| and for translators to help the public understand and follow
| the conversation in real time.
|
| We would all benefit from a greater understanding of all
| aspects of science, and from the increased trust that come with
| understanding.
| dr_dshiv wrote:
| Try the field of industrial design engineering / human-
| centered design / HCI. Huge growth and openness.
| deoxyribo wrote:
| I love what you're trying to accomplish here! I encourage you to
| experiment aggressively and try to focus on testing the biggest
| risks, rather than prioritizing the easiest things to make
| progress on but whose success or failure won't impact your
| overall odds of achieving your goal.
|
| One thing which I've noticed in your response to other comments
| is that your plan for controlling bad behavior by participants in
| your system often boils down to either having a strong culture
| means no one will behave poorly, or that voting and crowd
| consensus is an effective system for both motivating good
| behavior and punishing bad behavior. This probably works fine as
| long as (1) the community is very small such that everyone
| actually knows everyone else, or (2) there isn't much actually at
| stake. You reference Stack Overflow and Wikipedia - these are
| examples where not much is at stake. If accumulating points and
| reputation in either of these systems would determine who gets
| tenure and the culmination of lifelong career ambitions, then I
| wonder if those systems would be as robust to bad actors. A
| white-hot risk I think you should focus on to experiment is that
| peer-reviewed academic science is not a friendly collegial system
| but an aggressive and high-stakes game where people have very
| very big incentive to game the system. The current system has
| tons of problems, but it does function in the face of
| participants who would like to cheat if they could. The big
| influential editors and reviewers act like a leviathan in the
| Hobbesian sense which causes many problems but also solves many.
|
| I encourage you to think about how you could solve some of these
| potential issues that the current system _doesn't_ have so much,
| particularly around preventing various forms of bad behavior
| which are easily caught and punished in a more hierarchical
| system like the current one, but which a more crowd-sourced
| system might fall prey to. Things like collusion rings to upvote
| each others' papers, using lay popularity or scientific fame to
| overwhelm legitimate complaints about issues with a paper, even
| bot nets or paying to win by farming upvotes - all things which
| current journals and peer review are basically immune to. People
| will try this with their careers and livelihoods on the line in a
| way Wikipedia editors or Stack Overflow users just wouldn't care
| to.
|
| These issues can be overcome, but for what it's worth I think
| this is the hardest part of what you're trying to accomplish, and
| more important for you to focus on than the tech stack or getting
| a first few papers published on your setup. It will be hard to
| simulate, but I think you should focus on how to test the
| behavior of a scientist who is desperate and has their back
| against the wall in the publish or perish game, and how you can
| design to reward good behavior and prevent bad behavior in the
| worst case, not just in the best case or even the average case.
|
| Very cool effort. If you pull this off the positive externalities
| could be huge.
| [deleted]
| Kydlaw wrote:
| Having left academia after my PhD, but still doing research in
| the private sector, I hate the scientific publishing industry
| wholeheartedly. It is something of the past that is definitely
| keeping us, as a society, from making progress.
|
| There are more and more initiatives to create new tools around
| the literature (Semantic Scholar, ResearchRabbit and stuff like
| that) but less around moving he publishing industry as a whole.
|
| (I'm not affiliated with the previous organizations though I
| think Semantic Scholar have an API for peer review matching that
| might be useful in your case).
|
| If you need contributors I'd be interested.
| tensor wrote:
| I don't think scientific publishing is perfect, but I strongly
| disagree that it's somehow holding back society. T
|
| he number one thing that is holding back society in my opinion
| is that the vast majority don't think science is important or
| useful, and even worse, think that their opinions are somehow
| the same as scientific knowledge. Even convincing people in
| industry that basing decisions on data can be hard, let alone
| convincing them to use something approximating the scientific
| method.
|
| In short, if people wanted to read science papers but couldn't,
| I might agree with you, but people don't even want to.
| murphyslab wrote:
| > The platform splits pre-publish peer review from post-publish
| refereeing. Pre-publish review then becomes completely about
| helping authors polish their work and decide if their articles
| are ready to publish. Refereeing happens post-publish, and in a
| way which is easily understandable to the lay reader, helping the
| general public sort solid studies from shakey ones.
|
| The author doesn't seem to understand the "referee" role of
| (associate) editors and reviewers. The core idea is about
| gatekeeping of a journal title's self-set publication limit in
| favour of articles that represent a substantive advancement of a
| field or concept.
|
| However this itself is a problem for many fields of research,
| since it's a core factor underlying the failure to engage in
| replication studies and to check if results are valid. Moreover
| this would tend to encourage researchers with small but useful
| results to shelve them and NOT publish, since it will create a
| negative feedback mechanism for such contributions:
|
| > Up votes increase the paper's score and grant the authors
| reputation in the tagged fields. Down votes decrease the paper's
| score and the author's reputation in the tagged fields.
|
| Part of this problem is because, like any site which uses an
| upvote or downvote function, the button will be used to express
| things other than the intended meaning. "Does not contribute"
| quickly becomes "I don't like X" (even if it is methodologically
| valid).
| CJefferson wrote:
| I find the idea that papers are judged by a simple up-vote/down-
| vote terrifying.
|
| Popular academics will post their new paper on Twitter, which
| will immediately lead to a round of upvotes from their friends
| and followers. People who are currently out of favour will be
| downvote-bombed.
|
| I see no suggested way of stopping this -- and I hope noone
| thinks the answer is academics are "above that".
|
| I'd suggest changing from "people may leave a comment if they
| wish when they downvote" to "any vote requires at least 200 words
| of justification". In general I think it's important reviewing
| requires thought and justification.
| dbingham wrote:
| This is helpful feedback, and requiring responses with votes
| was already something we were pondering. And I'm increasingly
| leaning in that direction.
|
| One thing that wasn't covered in the blog post was the
| intention for downvotes to cost reputation, same as on SE right
| now. That should help disincentivize brigading to some degree.
|
| The field system may also help counter that tendency to some
| degree. By choosing which fields you publish in, you can choose
| the subset of people able to review and vote.
|
| But there is also a degree to which we will try to counter this
| with culture. The cultures of internet communities can be
| extremely stick and resistant to change, both good and bad.
| Just think about Reddit vs StackExchange. Roughly similar
| systems (though with very different goals and guidelines), very
| different cultures. If we can establish a strong culture on the
| platform from the get go - a culture of low ego collaboration
| and seeking good science above all else, that can potentially
| help counter some of the worse tendencies of wider academia as
| more and more of it moves into the platform. At least in the
| early days, before we can afford active moderation. In the long
| run, active moderation, and tools for community moderation, can
| help as well.
| ta123456789 wrote:
| "The ultimate effect of pay-to-play is that the traditional peer
| review and refereeing process has broken down. If a dishonest
| researcher gets rejected from a reputable journal, they can take
| their paper to a pay-to-play journal and have it published there.
| With over 10,000 academic journals, it's impossible for the lay
| public to track which journals are reputable and which are not.
| As far as the public is concerned, a published paper is a valid
| paper."
|
| Ok, but I take some issue with this.
|
| Most scientific papers are not intended at the general public,
| but at experts working in the specific area that the paper
| covers. Also, often these papers are not even understandable by a
| phd outside the area, let alone a lay person.
|
| Which journals and authors are reputable are known to experts in
| the area. "Pay to play" papers published in obscure journals will
| not get much attention or be taken seriously.
| throwaway1777 wrote:
| You are correct they shouldnt be intended for public to take
| seriously. I think one issue is that the news media frequently
| spins these articles so the public ends up needing to decipher
| the accuracy.
| dbingham wrote:
| So here's the experience driving that observation: I'm adjacent
| to a number of anti-vax communities and during the pandemic
| (and before) there were PDFs of "1000 papers showing why
| vaccines are dangerous" being passed around. Honestly, snake
| oil salesmen don't even have to go through pay to play, they
| can just self publish and the effect is the same. If it looks
| real enough, a lot of people don't have the background to tell
| the difference. They look at the abstract and the conclusions
| and go from there.
|
| If we're going to beat that kind of misinformation, the real
| stuff needs to be open and available so that translators can
| help people understand it. There were people in academia doing
| this on social media during the pandemic, but we need many,
| many more of them. There are a lot of people outside of
| academia who have enough of a background that they could help -
| if only they had access.
|
| Edit to add: Also, I've heard from some folks in academia that
| they've been bitten by open access journals that started out
| well intentioned and turned into pay to play with bad
| reputations. And that in some fields, it is hard to track which
| ones are good and which ones are going bad.
| Test0129 wrote:
| Agree. When I was in graduate school the "bad journals" were
| well known. The pay to play journals are very cleverly
| disguised, but it's extremely hard to accidentally publish to
| or cite from one of these journals.
|
| There are many, many problems with the almost cult-like
| atmosphere of reviewers in good journals but pay-to-play isn't
| one of them.
| periram wrote:
| Awesome effort. I really really hope this makes at least a minor
| dent in the scientific publishing deadlock.
|
| My 2 cents: Please be open to suggestions. Please involve more
| people, who you feel comfortable working with. You have solved
| multiple independent problems of decentralized trust, voting etc.
| Please try to be agnostic to a specific solution.
| dbingham wrote:
| Will do my best! That's part of the reasoning behind early ask
| for feedback and my eagerness for it :) Trying to get people
| involved as early in the process as possible to get more
| perspectives.
| istillwritecode wrote:
| Reviewing is not the cause of the economic problems in academic
| publishing. The economic problems arise from the car at of copy
| and production editing. That's where the money goes. See
| https://f1000research.com/articles/10-20
___________________________________________________________________
(page generated 2022-08-07 23:01 UTC)