[HN Gopher] Guidelines on how to be a scientific sleuth released
       ___________________________________________________________________
        
       Guidelines on how to be a scientific sleuth released
        
       Author : crescit_eundo
       Score  : 42 points
       Date   : 2025-06-17 17:12 UTC (5 hours ago)
        
 (HTM) web link (osf.io)
 (TXT) w3m dump (osf.io)
        
       | crescit_eundo wrote:
       | From the project's introduction:
       | 
       | "Anyone can do post-publication peer review. Anyone can be a
       | steward of the scientific literature. Anyone can do forensic
       | metascience. Anyone can sleuth. However, investigating the
       | integrity of the published scientific literature often requires
       | domain-specific knowledge that not everyone will have. This open
       | source project is a collection of guides written and maintained
       | by publication integrity experts to distribute this domain-
       | specific knowledge so that others can participate in post-
       | publication peer review."
        
       | btilly wrote:
       | I consider this an extremely important project.
       | 
       | One of the most reliable cognitive biases that we have is that
       | cognitive dissonance prevents us from taking feedback on that
       | which is important to us. Instead we convince ourselves that
       | we've become good at that which we care about, and reject all
       | evidence that we need to improve. The more we care about it, the
       | more we fall into this trap.
       | 
       | There are ways to reduce how often we fall into this trap. For
       | example https://blog.codinghorror.com/the-ten-commandments-of-
       | egoles... is a set of rules that many good programmers follow to
       | avoid developing an ego about being a good programmer. This
       | allows them to accept feedback from reality, and become better
       | programmers over time. By contrast, programmers who think that
       | they are good usually top out pretty quickly, and are usually
       | pretty bad.
       | 
       | Unfortunately we always have a pull to believing that what we've
       | done is valuable, and then falling into this trap. This causes
       | entire fields of study, including scientific fields, to
       | frequently fall into a self-reinforcing overconfidence in what
       | they believe that they have shown. Which consistently results in
       | missing their own errors, and rejecting contrary evidence.
       | 
       | Post-publication peer review by itself does not fix the problem.
       | But it is a necessary feedback loop that is required for any
       | meaningful fix. And the existence of post-publication peer review
       | helps those not in the field to learn which fields should be
       | trusted, and which not.
       | 
       | Therefore there is a lot of value in having more, and more
       | effective, post-publication peer review. Even though we'll still
       | have entire fields of science who are overconfident of their bad
       | research.
        
       ___________________________________________________________________
       (page generated 2025-06-17 23:01 UTC)