[HN Gopher] Scientific integrity in a climate of perverse incent...
___________________________________________________________________
Scientific integrity in a climate of perverse incentives and
competition (2017)
Author : zoid
Score : 92 points
Date : 2022-02-21 08:34 UTC (14 hours ago)
(HTM) web link (www.ncbi.nlm.nih.gov)
(TXT) w3m dump (www.ncbi.nlm.nih.gov)
| silicaroach wrote:
| This perfectly describes the current state of academia. The
| belief in treating academia like a business to promote
| 'efficiency' and 'accountability' leads to all the ills you see
| in business. The bottom line is that you should be extremely
| distrustful of anything aggressively promoted as 'scientific
| consensus'. If an idea is successful, whether correct or not,
| then as in business people will blindly support it to 'get a
| piece of the pie'. As an example, machine learning is replete
| with incremental, incorrect, and false papers all riding on the
| success of a few genuine results. The most successful academics
| are not the smartest or best researchers, they are those best at
| marketing their results.
| ComradePhil wrote:
| In reality, there is no such thing as "consensus" in any expert
| field, not even in maths or physics, not even when it comes to
| fundamental theorums, let alone in leading edge or new fields
| of study. So, if someone claims there is "consensus" among "9
| out of 10" or "97%" or similar, it's a tell that there is
| something else in play.
| jpeloquin wrote:
| > If an idea is successful, whether correct or not, then as in
| business people will blindly support it to 'get a piece of the
| pie'.
|
| I wonder whether articles like this are driven in part by the
| same copycat effect described above. People love to complain
| about their workplace. Although the article identifies real
| problems, it is lean on evidence for the magnitude of their
| impacts. Many of the citations are to other opinion pieces.
| It's certainly worth improving things where we can, but the sky
| is not falling either.
|
| For example, the abstract leads with "If a critical mass of
| scientists become untrustworthy, a tipping point is possible in
| which the scientific enterprise itself becomes inherently
| corrupt and public trust is lost, risking a new dark age with
| devastating consequences to humanity." Which sounds serious.
| But based on articles posted to Hacker News previously, 1-10%
| of scientists (excluding countries with direct cash payments
| for accepted articles) fabricate or falsify data, which is a
| typical percentage of bad actors in most groups. The other 90%
| still appear able to serve as quality control in the usual way.
| People can tell if an article is likely to be b.s. with about
| 70% accuracy based only on its description, not even reading
| it. I think any defects in science feel more severe than they
| really are because science has been so successful that our
| expectations are now sky-high.
|
| I do emphatically agree that promoting altruistic and ethical
| norms should be a focus of scientific institutions. Major
| discoveries will arise from curiosity and serendipity, as they
| always have, they cannot be predicted in a proposal's Gantt
| chart.
| [deleted]
| DangitBobby wrote:
| How do I square this with the knowledge that some groups benefit
| by discrediting legitimate scientific consensus, as has happened
| numerous times in the last century? I'll share my own FUD [1]:
|
| > The statement reading "Indeed, delving deeper, 34% of
| researchers self-reported that they have engaged in "questionable
| research practices," including "dropping data points on a gut
| feeling" and "changing the design, methodology, and results of a
| study in response to pressures from a funding source," whereas
| 72% of those surveyed knew of colleagues who had done so
| (Fanelli, 2009)." should have been written as, "Indeed, delving
| deeper, up to 34% of researchers self-reported that they have
| engaged in "questionable research practices," including "dropping
| data points on a gut feeling" and "changing the design,
| methodology, and results of a study in response to pressures from
| a funding source," whereas up to 72% of those surveyed knew of
| colleagues who had done so (Fanelli, 2009)."
|
| Guess we can't trust the authors of this paper!
|
| 1.
| https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5564049/?report...
| qchris wrote:
| (2017)
| zivkovicp wrote:
| My guess is that it's still relevant. :D
| Proven wrote:
| ganzuul wrote:
| There may be technical remedies if not cures for this channel
| noise issue. A community which grew out of BOINC got involved
| with distributed ledgers and independently invented many concepts
| which are hyped today. - They get no attention because they are
| non-profit, but they stand poised to reinvent the basic
| infrastructure underlying peer review and to with it upend
| conventional systems of governance because the basic problem they
| are facing (as per the topic) is the same. I think it's a matter
| of time before that project branches or merges with more visible
| distributed grids.
| mistrial9 wrote:
| lots of solid research work on distributed consensus, layered
| voting, and "deep democracy" existed 20 years ago, on paper.
| Yet we got a decade of Facebook and spy-oriented DNS tricks,
| for the masses. I am not certain that that technical excellence
| you describe means very much at all, by itself. To further
| adoption in the field, there are multiple and well-known steps
| to implementation. Do these BOINC academics want that? not
| clear at all ..
|
| ref: Diffusion of Innovation studies
| https://en.wikipedia.org/wiki/Diffusion_of_innovations
| fouc wrote:
| General question about incentives..
|
| How do incentives get fixed? Do we have any social systems
| engineers that focus entirely on designing or adjusting
| incentives to encourage social systems to organically produce the
| desired outcomes?
|
| Is there any incentives-first systems thinking going on?
| derbOac wrote:
| No one wants to talk about this (or at least few), but in my
| experience, so much is being fed from the federal government,
| at least in the US, that changes at that level force massive
| changes system-wide. Without meaning to come across as
| reactionary, most of US academics and its problems are being
| driven by money flows from the federal government to
| universities via grant mechanisms.
|
| I think like a lot of major problems, it will require a lot of
| changes coming from a lot of places.
|
| One massive one, though, in my opinion is serious, significant
| changes in grant funding processes. There need to be big
| changes to how grants are awarded and funding decisions are
| made, and how those grants are actually funded. Moving to more
| of a lottery-style system (where everything above, say the
| 50%tile are entered into a lottery), and funding individuals
| are examples.
|
| I also seriously believe indirect funds need to be cut. In the
| very least least indirect funds need to be line-item justified,
| rather than being based on some arbitrary percent cut or
| however they're negotiated now, with independent audits of the
| indirect funds to determine whether universities are asking for
| more than they're actually cost.
|
| Some universities get around the indirect fund incentive
| structures in creative ways. For example, one place I know of
| takes all the indirect funds and redirects it toward
| university-grant support services at the university, so none of
| it goes into other activities, and it doesn't go to specific
| departments. But even that seems insufficient.
|
| A lot of this is going to require other things, like trimming
| administrative bloat, democratizing university decision-making,
| punishing superficial promotion and tenure criteria, and
| increasing transparency about why universities are making
| decisions that they are. This is just the tip of the iceberg.
|
| I have no idea what's going to happen. Former federal agency
| heads have compared it to the pre-2008 financial bubble, and
| that's what it feels like to me. But how that bubble pops I
| have no idea. I could see the government coming in and cutting
| indirect funds; I could see something like indisputable proof
| of COVID being the result of a lab leak maybe causing a public
| outcry over what's going on; or I could see people just not
| going to grad school anymore and things drying up. Or maybe it
| will just get exponentially worse for awhile.
| Matumio wrote:
| This quote from the article sums it up:
|
| > "When you rely on incentives, you undermine virtues. Then
| when you discover that you actually need people who want to do
| the right thing, those people don't exist." --Barry Schwartz
|
| I'd say you need to reinforce psychological traits in the work
| environment, like following your curiosity for the sake of it
| (watch others do it), or feeling guilty about publishing
| something that you know may mislead the readers (watch your
| supervisors correct such failures, instead of teaching you how
| to trick your paper into a high-impact journal). De-emphasize
| gate-keepers, the quantitative measurements (like citation
| count), financial pressure to perform. All of those will be
| gamed if they are too important.
|
| Then build a reputation (as an institution) to work that way, I
| guess. This will give publications from that institution a
| higher credibility (long-term). The cost of being mislead by
| the claims of every paper you read is simply too high.
| [deleted]
| fristechill wrote:
| >"When you rely on incentives, you undermine virtues. Then
| when you discover that you actually need people who want to
| do the right thing, those people don't exist." --Barry
| Schwartz
|
| Quite so. And things may be worse than that. It may be that
| both incentives and virtues are unreliable. For example, the
| famous day care study mentioned in _Freakonomics_ casts doubt
| on the utility of incentives when it comes to social problem-
| solving:
|
| https://sites.google.com/site/cvhsbahm/economics/econ_calend.
| ..
|
| Moreover, my guess is that virtues themselves may also be
| unreliable because they're are about outward behaviour, which
| is often inherited and not explicitly understood, and which
| may not be passed on.
|
| Jacob Bronowski, scientist and author of _The Ascent of Man_
| , identified the primary scientific virtue as what he called
| the _habit of truth_. This is to rigidly tell the truth about
| all things, both in private and in print, including about the
| minutest details, in one 's scientific work.
|
| It is a matter of opinion, but it seems that the habit of
| truth no longer pervades the scientific enterprise, now fully
| professionalized and bureaucratised.
|
| Perhaps it was lost because it was only a habit. Whereas the
| _love_ of truth, beauty, and so on, are spiritual values:
| modes of being rather than habitual outward behaviours. Which
| may explain why (according to Ed Dutton and Bruce Charlton),
| many 20th century scientific geniuses were first generation
| atheists. They inherited a reverence for truth and reality;
| they were able to make important scientific progress, but
| their ardour could not be sustained beyond a generation or
| two.
|
| https://geniusfamine.blogspot.com/
| yldedly wrote:
| I started a PhD believing that scientists still held on to
| the habit of truth. The gradual realization that modern
| science rewards bullshit artists, and that great research
| happens in spite, not because of today's science culture,
| is one of the most bitter disappointments of my life. I
| feel robbed of one of the main reasons to be proud of being
| human.
___________________________________________________________________
(page generated 2022-02-21 23:01 UTC)