[HN Gopher] Scientific progress despite irreproducibility: A see...
___________________________________________________________________
Scientific progress despite irreproducibility: A seeming paradox
Author : nabla9
Score : 32 points
Date : 2021-08-18 18:58 UTC (4 hours ago)
(HTM) web link (www.pnas.org)
(TXT) w3m dump (www.pnas.org)
| baron_harkonnen wrote:
| > Proving the claims of rapid progress would be inordinately
| difficult and beyond the scope of this contribution.
|
| That seems like it's an important part of this issue. If we know
| that there are massive problems with irreproducibility and we
| just have to go on an assumption that there has been "rapid
| progress" then it seems like the most obvious solution to the
| "paradox" is to question that progress as well.
|
| This is similar to how in the 90s everything was labeled "healthy
| and fat free!" which we know resulted in food filled with sugar,
| then making the claim "the paradox of healthy food despite high
| sugar". It might be worth calling into question the claims of
| healthiness given the fundamental conditions that produce health
| aren't present.
|
| Certainly there are places that everyone will agree we've seen
| progress, but all of these places seem to be where scientific
| progress is closely tied to commercial application.
|
| One of the best examples that I think everyone will agree has
| been remarkable has been the storage capacity of batteries. While
| much of this progress surely starts in the labs, the true measure
| of progress isn't in papers it's commercial applications. It
| frankly doesn't matter at all if the academic research behind
| battery technology was good or not, because we know these
| batteries work and are in fact smaller. If you cellphone weighed
| 10 lbs and ran out of energy in 45 minutes you wouldn't care
| either way what the research said.
|
| However the answer to this "paradox" isn't just that the market
| is some force for testing what is real. Anyone who has worked
| long enough in machine learning knows a fair amount of bullshit
| not only exists in papers but in products as well.
|
| A better explanation for me is that we live in an age of
| unprecedented economic and high energy intensity activity. You
| generate 160,000 TWh of power you're going to see a lot of
| wonderful things that look like progress. Some real, some
| illusions, but assuming that science is really the basis for all
| of this is a fairly large, and unchecked in this article,
| assumption. Lots of the scientific progress we've made in the
| last few decades has happened outside of an academic research
| lab, and assuming "science" is the cause might be a bit naive.
| enkid wrote:
| I agree with almost all that you say, bit I disagree with the
| implication that progress in commercial enterprises isn't also
| a form of scientific progress. I totally agree that the
| ultimate measure of scientific progress is how much of that is
| accessible to wider society. That doesn't have to be in a
| published journal, it's anything that allows us to know about
| the world and leverage that knowledge for the better.
| EverywhereTrip wrote:
| The problem isn't irreproducibility in "science".
|
| It is irreproducibility in a few fields. Most notably, nutrition,
| psychology, and economics.
|
| These are all fields which study humans. The study of humans is
| far more fraught with bias and ideology. Humans are also
| independent decision making agents and behave in a way that atoms
| do not.
| elisharobinson wrote:
| why stop there modern medicine is also full to the brim with
| non tested hypothesis. i,e only recently have we seriously
| looked at the effects of saline and the placebo effect is well
| documented phenomenon.
| adamisom wrote:
| Funny you mention placebo effect. It actually is probably far
| less of a thing than popularly believed see
| https://slatestarcodex.com/2018/01/31/powerless-placebos/
|
| From one point of view that just proves your point more, from
| another, less. If placebo effect is tenuous then all else
| equal that's a good sign for the rest of medicine... but in
| fact lots of medicine is tenuous. The landmark "Most
| Published Research Findings Are False" was looking at medical
| findings iirc.
| derbOac wrote:
| Surveys suggest it's not limited to a few fields. You might be
| right about fields studying humans though.
| tomp wrote:
| Is this even true? Most of the progress is in "hard sciences" -
| physics, cellular biology, computational biology, genetics ...
| where experiments (and presumably reproducibility) is just fine.
| microtherion wrote:
| As for biology, I recommend following e.g. the work of
| Elisabeth Bik to see the hair raising amount of fraud going on,
| and getting published for years (or, if you like extra spice in
| your diet, you can follow Leonid Schneider).
|
| For physics, consider how difficult it is to settle the
| controversies around something like the EmDrive, even among
| experimenters with solid reputations and impeccable
| professionalism.
| querez wrote:
| Counterpoint: I have a friend who works in a biomed lab where
| results would regularly just be made up. If a lab culture takes
| 5 years to grow, reproducibility is more of a theoretical than
| a practical matter. Also, not every reviewer has a particle
| accelerator in their back yard. Or the money to reproduce large
| computational models.
|
| A lot of published results only pass peer review because you
| essentially trust the authors not to have made up their
| numbers.
| epistasis wrote:
| As somebody in computational biology, I think it's important to
| note that a ton of cellular biology and even computational
| biology is "not reproducible." This could be anything from bad
| documentation of how something was run (computational
| scientists are often _terrible_ at lab notebook culture), to
| having a model system that lived for a while and no longer
| exists in the form that initial discoveries were made with. Or
| it could be that there was just that one batch of reagents that
| reliably reproduced a phenomenon, and all future orders of the
| reagents no longer get the same phenomenon, and whatever the
| difference was is lost to time.
|
| But I would claim that especially in biology, though this is
| less than ideal for writing up near little explanations of
| reality, it is still extremely useful for understanding what's
| going on. Scientific papers aren't meant to be ever-lasting
| truth, like a textbook. They are communications amongst
| specialists about "look here something cool happened that may
| be useful to you too." It is only through lots of work that a
| phenomenon can be established as widespread in biology, and
| sharing information before doing 10 years of work helps
| accelerate everything.
|
| This is why I roll my eyes at complaints like that one about
| "foundational cancer research papers not being reproducible."
| It was written by scientists in industry who wanted to take a
| new paper and develop an entire drug program around something
| particularly novel and surprising. Sure, that would be ideal if
| it worked, but the scientific literature is a lot more than a
| catalog of ideas ready for commercialization. Scientists in
| practice understand the limitations of taking a journal paper
| as gospel. They always try to get something working in their
| own hands before basing a lot of research off another paper.
| derbOac wrote:
| A lot of the biomedical sciences has similar problems with
| reproducibility. A survey found similar problems in
| pharmacology (I think there was one article quoting someone
| saying the pharmaceutical companies they were familiar with
| internally budgeted for about 2/3 of published academic
| articles to be false) and oncology, and others.
|
| Whatever you might have to say about research in psychology,
| it's also the field primarily turning the microscope on itself.
| This is part of a tradition in the field -- modern meta-
| analysis has its origins there.
|
| I'm less familiar with physics but there's a lot of problems
| with reproducibility in many fields.
| tomp wrote:
| > A lot of the biomedical sciences has similar problems with
| reproducibility.
|
| Indeed, which is why I specifically singled out cell biology.
| It's the part of medicine that's closest to physics.
| Oncology, pharmacology etc. study humans/whole bodies, so not
| only are the experiments more expensive to run and much more
| noisy, there's also all kinds of ethical issues. Most of
| these aren't there, or are at least reduced, when dealing
| with just cells (I didn't expect "cell cultures take a long
| time to grow" and "the chemicals used are non-reproducible"
| issues that sibling comments pointed out, so I guess not
| _quite_ physics)
| analog31 wrote:
| In my view, reproducibility is neither strictly necessary nor
| sufficient for scientific progress. I'm sure if you went back
| through the literature on something like electromagnetism, you
| would find results that fail to replicate, yet the theory of
| electromagnetism if applied properly is remarkably robust.
|
| Scientific results can be strengthened by replication, but that's
| just one thing, and if that's all you do, then you end up with a
| science that does nothing but generate independent factoids.
|
| On the other hand, robust science tends to look at a particular
| phenomenon from many different angles, and manages to connect
| multiple results together into a framework that can survive
| retracting individual studies without collapsing. This is how
| electromagnetism developed.
|
| Sciences that are in the factoid phase are not necessarily junk.
| Discovery of a psychological "effect" is perfectly scientific and
| interesting. But some sciences have barely progressed beyond the
| factoid phase. And if those sciences are also plagued by
| irreproducibility, then they may embrace scientific methodology
| without producing a useful scientific knowledge base.
| atty wrote:
| As a physicist, I think how we normally describe what you're
| talking about is that there must be a robust and explanatory
| theory that goes along with experimental measurements. The most
| fruitful areas of research are the ones where theory and
| experiment exist on approximately the same level of what they
| can predict or measure. These strong explanatory theories are
| the things that generate truly robust, reproducible
| experiments.
|
| In cases where there is significant differences in the maturity
| between theory and experiment, you either end up with theorists
| playing games with math, making up tons of unconstrained
| theories (string theory, for instance), or in the other
| direction you end up with experimentalists measuring anything
| and everything they can imagine, half of which is probably not
| theoretically enlightening, waiting for theorists to constrain
| their space of possible experiments to potentially fruitful
| paths.
|
| Obviously this is harder for fields that don't have models as
| robust as we do in physics, but I'd guess the same phenomenon
| happens everywhere.
| LatteLazy wrote:
| Wait, what electromagnetic results are not reproducable!?
| enkid wrote:
| I think the parent poster is saying some EM experiments
| weren't initially replicated because of the peculiarities of
| the set up, etc. Another example is that some people couldn't
| see the Galilean moons initially. This made initial report of
| them somewhat questionable, as some observed them and some
| didn't.
| mensetmanusman wrote:
| One example: What would progress in psychology even look like?
| More people than ever are medicated for mental illness, seems
| like the opposite of progress.
| Tenoke wrote:
| Clinical Psychology is just one part of the field of Psychology
| but even there can be progress in many directions - better
| diagnosis, less reported issues due to better prevention, less
| side-effects from treatments, or just pure metrics like higher
| life satisfaction, less suicides etc.
| 542458 wrote:
| I don't think number of diagnoses is a great metric for or
| against the progress of psychology. We could outlaw psychology
| and have zero diagnoses, but that wouldn't be progress. If
| we're criticizing psychology for not "curing" people I think
| that's a bit unfair - people may not be cured, but their
| conditions can be managed. I personally would love to be cured
| of what ails me, but that doesn't mean I'm not grateful for the
| medications and therapy that help me live a more "normal" life
| than would otherwise be possible.
| _Microft wrote:
| The field of psychology is far larger than just clinical
| psychology. So much larger that I would recommend looking it up
| on Wikipedia instead of listing things here.
| nomel wrote:
| With the assumption that their problems are real, what's the
| alternative? Surgery?
| inter_netuser wrote:
| Everyone who reads scientific papers as a part of their job know
| a good chunk, if not the majority, of papers are fluff, and not
| only in social sciences.
|
| I've been a reviewer for a journal and we'd always recieve a good
| chunk of marketing whitepapers dressed up in just enough jargon
| to be published.
| miga wrote:
| It has been conjectured before, that probably "most published
| research findings are false"
| https://en.wikipedia.org/wiki/Why_Most_Published_Research_Fi....
| But science is a body of the findings that are confirmed many
| times, harder and harder to falsify.
| api wrote:
| If you are descending a learning gradient you don't need every
| step to be perfect. You just need the general progress to be
| along the gradient. If science is even a little bit more right
| than wrong, theoretically it will follow the gradient over time
| (with some wiggles).
|
| I don't think anyone would argue that more than 50% of scientific
| publications or findings are flawed.
| oerpli wrote:
| Some (e.g. Marc Andreessen:
| https://richardhanania.substack.com/p/flying-x-wings-into-th...
| ) say, 90% are either wrong or useless. It's roughly what I
| would guess as well.
| mirker wrote:
| Citations follow some power-law type of distribution, so the
| 10% (or whatever) of useful ones are basically the high
| impact papers, anyway. It would be more surprising if
| citations and usefulness were anti-correlated in some way.
|
| Back to the "gradient" analogy, in this case, papers have a
| feedback mechanism to suppress less useful papers. Science is
| not a random flurry of results mashed together.
| wolverine876 wrote:
| Every human institution is flawed, which imposes costs, but costs
| needn't bankrupt the enterprise. Every business is flawed, but
| some succeed (YC could tell you something about that). All
| software is flawed. The only solution - the only means to success
| - is to minimize the flaws and to find solutions that deliver
| positive returns on the investments. Democracy itself is very
| flawed, but works; so does science. (That's not an excuse for the
| flaws or a reason to accept them.)
___________________________________________________________________
(page generated 2021-08-18 23:00 UTC)