[HN Gopher] Never Trust the Science - On the need to identify bi...
___________________________________________________________________
Never Trust the Science - On the need to identify bias & interpret
data yourself
Author : Luc
Score : 22 points
Date : 2026-03-18 11:20 UTC (11 hours ago)
(HTM) web link (adam.rochussen.xyz)
(TXT) w3m dump (adam.rochussen.xyz)
| afpx wrote:
| How about a big vetted database like arxiv of all hypotheses, all
| proposed experiments to test them, and all experimental results?
| lithocarpus wrote:
| Vetted by who?
|
| To be clear I'd be very much in favor of scientific studies and
| their data having to be publicly available.
|
| But on any controversial area, which is most of the areas
| anyone cares about, there will be 2+ sides of the issue and any
| vetting body will be compromised to some degree for one of
| those sides.
| tracker1 wrote:
| That's the rub, isn't it... who watches the watchmen? In
| times past, journalism at least had the veil of impartiality,
| but modern journalism is far more of an editorial activist
| activity than simply answering the 6 W's of a given story.
|
| I'm not sure it was ever actually much better... and it may
| just be my pessimistic Gen X nature. But I've personally seen
| too many misrepresentations about too many studies where the
| body and available data in fact don't match the headlines or
| the numbers themselves are deceptive in a way that is much
| less significant than represented.
|
| 200% the risk of X... when in sample A of 10000, 1 had X, and
| in sample b of the same size, 2 had X... while it's a real
| relative stat, the absolute values are all but meaningless in
| context.
| thephyber wrote:
| Yellow journalism existed generations before you and I. The
| institution was always sullied by the worst and has always
| contained some of the most dogged pestering fact finders.
|
| It's not even clear that journalists of the 1960s-1980s
| were as impartial or brutally honest as we remember. That
| is most likely a halo effect from having a few highly
| trusted very visible personalities (eg. Walter Cronkite),
| but even they were slow to realize (by a decade) how much
| of a morass the Vietnam War was.
| habinero wrote:
| arxiv is an open-access journal that checks for spam. It is
| very much not "vetted" lol
| lunatuna wrote:
| Ya, I want it even bigger. All commercial claims should be
| accessible for your own determination. Fastest, biggest,
| longest, widest, shortest, most liked, doctor recommended, any
| empirical claim must have the data used and calculations to
| make the claim available for examination. Data storage is so
| cheap now. I don't see it as a dent to anyone's profit.
| RcouF1uZ4gsC wrote:
| And if a result is surprising to you, you should trust it less
| and look into it more deeply.
|
| And if you do so, one of two good outcomes will hopefully happen:
|
| 1. You find the result is bogus
|
| 2. You learn something new and update your internal model of the
| world.
| thephyber wrote:
| This ignores a huge blind spot that humans have: confirmation
| bias.
|
| There is no accurate heuristic which is a good short.
| like_any_other wrote:
| > For example, failing to control for obvious confounders in
| observational data is likely to produce biased results. If we
| like the direction of this bias, we can do less adjustment for
| confounders.
|
| For example, the study showing that having a white doctor
| increased mortality of black babies didn't correct for birth
| weight - once that was done, the effect disappeared (and media
| interest waned): https://www.wsj.com/opinion/justice-jacksons-
| incredible-stat...
| _aavaa_ wrote:
| > Finally, if you want to simply know which Science(tm) you can
| trust, I'd recommend finding and following individuals who
| repeatedly demonstrate competence in statistical methods and
| scientific interpretation.
|
| So like, the scientists themselves?
|
| > If in doubt, read the study critically yourself.
|
| I cannot believe the author manages to say this with a straight
| face. "Hey you average person (with _maybe_ a college degree), go
| read the original academic paper yourself. Doesn't matter that
| you don't have the background, struggle with basic math (much
| less statistics), can't evaluate the claims, and don't know which
| questions to ask."
|
| The age of the polymath is long dead, we're living in The Great
| Endarkenment. You trust your pilot to do their job, you trust
| your civil engineers with the bridge you driver over, and the
| mechanical engineers with the controlled explosions happening in
| your car, but when it comes to cutting edge scientific articles,
| here is where you, average Joe, will be able to know better that
| the experts in the field who specialized in this and do it every
| day.
| like_any_other wrote:
| > You trust your pilot to do their job
|
| In this case, the "pilot" (the combined media and researcher
| science communication system) is deliberately steering the
| plane into the side of a mountain. A coin flip would do a
| better job. They've burned their credibility to the ground, and
| you're trying to repair it by invoking _other_ professions that
| haven 't done so.
| _aavaa_ wrote:
| Okay, I'll bite. Who exactly is "they"? How have they burned
| their credibility to the ground? And how does reading
| scientific papers yourself address this issue if in your
| telling it was created by people who are no better than a
| coin flip?
| thephyber wrote:
| This assumes that everyone is currently sufficiently informed
| enough to make the same expert observations about methodology
| affecting bias. This is flatly untrue for the vast majority of
| the population.
|
| And nobody has enough time or desire (or likely money to
| subscribe to the journals) to read the details of the papers and
| grok the nuances. Humans think in simple narratives for a reason.
|
| We shouldn't have blind faith in science, but we also shouldn't
| have to go back to first principles and do our own version of
| every experiment. The repeatability crisis is a thing and we know
| about it. P value hacking is a thing we know about.
|
| The problem described in the article is that we shouldn't believe
| headlines or short summaries created by writers who aren't
| incentivized to add the nuance. And nobody should believe a
| headline anyway - in addition to necessarily being lossy, for any
| for profit organization they are likely written by someone other
| than the writer and probably A/B tested for clicks.
___________________________________________________________________
(page generated 2026-03-18 23:01 UTC)