[HN Gopher] Many Researchers Using Same Data and Hypothesis Reve...
___________________________________________________________________
Many Researchers Using Same Data and Hypothesis Reveals Universe of
Uncertainty
Author : Ambolia
Score : 77 points
Date : 2022-10-17 10:31 UTC (12 hours ago)
(HTM) web link (osf.io)
(TXT) w3m dump (osf.io)
| dmichulke wrote:
| I wonder if this study would arrive at a different conclusion if
| it were conducted by a different team...
| pfortuny wrote:
| They address this in the abstract (or the introduction, I
| forget).
| btrettel wrote:
| I think this effect has been recognized in science for decades,
| even if it wasn't talked about much in print.
|
| Recently I thought about trying to combat this effect myself by
| using multiple different analysis methods. For example, in my
| case I'm thinking about trying multiple different regression
| approaches. My research is in the physical sciences, so I don't
| think in this case the overall conclusions will be that different
| because the general trend is clear from a scatter plot (often not
| true in social science...), but it'll still be an interesting
| exercise.
| adamsmith143 wrote:
| Should be careful to differentiate between "Social Science" and
| Hard Science. One of these is flooded with questionable papers
| that cannot be replicated and the other has fields where a p
| value of 3x10^-7 is required for a discovery to even be taken
| seriously.
| subroutine wrote:
| Setting the bar at seven sigma just means natural sciences are
| more biased towards our mathematical preconceptions about the
| world than accepting physical evidence. There's nothing
| inherently wrong with that - the same could be done in social
| science. It just means the null hypothesis (the accepted
| preconception) will be sustained more often. And more type-II
| errors will be made. A six sigma result means there is a one in
| a million chance of getting the result we observed from our
| experiment if our maths-based theory is correct. And what you
| are saying is that such a result is just "meh, our math still
| checks out".
| raincom wrote:
| There are many problems with this kind of hypotheses: (a) any
| hypothesis should predict/explain facts that are different from
| the original data/facts that the hypothesis accounts for;
| otherwise, it is ad hoc. (b) multiple competing hypotheses can
| account for the same set of facts. So, how to pick which
| hypothesis should one pick from the competing hypotheses? Again,
| here, Philosophy of sciences come to rescue. Pick a hypothesis
| that explains/solves more facts/problems. Here, the
| problems/facts should be novel, not the original ones.
| SamBam wrote:
| > Researchers' expertise, prior beliefs, and expectations barely
| predict the wide variation in research outcomes. More than 90% of
| the total variance in numerical results remains unexplained even
| after accounting for research decisions [including which
| statistical tests to perform] identified via qualitative coding
| of each team's workflow.
|
| They're saying that scientists, given the same data, will achieve
| widely different results, and that they can't even work out how
| they're getting these different results. That's surprising and
| concerning.
|
| > This reveals a universe of uncertainty that is otherwise hidden
| when considering a single study in isolation. The idiosyncratic
| nature of how researchers' results and conclusions varied is a
| new explanation for why many scientific hypotheses remain
| contested.
| Enginerrrd wrote:
| I've certainly got into the habit of reading the methods and
| results of social science papers that make headlines and what
| I've found has been horrifying. More than half of the time,
| there are serious, serious flaws involved in the data
| collection or the analysis. Even worse, I've frequently
| encountered situations where the underlying data really seems
| to point toward the opposite conclusion than the one made by
| the researchers who have really twisted and beaten the data
| into submission to get the conclusion they wanted.
| DontchaKnowit wrote:
| Yep, which is why I find the use of "science" as a political
| battering ram to be one of the most infuriating modern
| phenomena.
| jltsiren wrote:
| It's not surprising at all. There are always many arbitrary
| choices to make and many things that can go wrong if you try to
| make conclusions from data. Even worse if you try to automate
| making the conclusions. And because the data you have has often
| been derived from lower-level data, many arbitrary choices have
| already been made and many things have already gone wrong.
|
| On good days, I find it a small miracle that we can sometimes
| make justified conclusions from data. On bad days, it feels
| that making conclusions from data is storytelling, not science.
| a1369209993 wrote:
| > On good days, I find it a small miracle that we can
| sometimes make justified conclusions from data. On bad days,
| it feels that making conclusions from data is storytelling,
| not science.
|
| If you dig far enough down the rabbithole, you eventually
| find that the numbers are being pulled out of someone's
| proverbial ass. But pulling numbers out of your ass and using
| them to make a conclusion is a lot better than pulling a
| conclusion out of your ass and using it to make numbers.
| mellosouls wrote:
| _It calls for greater humility and clarity in reporting
| scientific findings._
|
| Hmmm, I think the variance in results is maybe a bit more of an
| issue in the softer sciences and the conclusions would be better
| worded to reflect that considering the context here, rather than
| implying they hold to a similar extent over science generally.
|
| There is no doubt human failings also challenge physics etc (see
| previous threads here) and humility and clarity is appropriate
| there also, but the significant problems highlighted in this
| study seem more particular to areas of research that have long
| been regarded as less sure due to the nature of their domain.
| uniqueuid wrote:
| Personally, I prefer the term science of soft reality.
|
| After all, the science is as hard as it can be, it's the object
| of investigation that's fuzzy and shifting and actively
| refusing to be measured ;)
| SuoDuanDao wrote:
| Does the term 'soft reality' make sense though? I don't
| subscribe to realism or materialism, but I understand the
| instinct to reduce the universe to just the measurable parts
| if the alternative is that the scientific method doesn't work
| at all.
|
| Granted, real science has error bars and there's no upper
| limit to how big those could get _in theory_ , but in
| practice there's an upper limit to where a scientific
| approach is better than just crafting a compelling narrative.
| mvcalder wrote:
| That's an unscientific conclusion. They say the effect (of high
| variation) is unattributable, so it's unfair to limit it to
| soft sciences.
|
| This could infect literally any human endeavor that generates
| measurements, hard sciences included.
|
| Or, as another poster observes, this result itself is affected
| by high variation and we end up in a Godel-ian loop of "this
| study is false" results.
| macintux wrote:
| Agreed, this seems like they chose a hypothesis destined to be
| forever unprovable.
|
| > they used the same data to independently test the same
| prominent social science hypothesis: that greater immigration
| reduces support for social policies among the public.
| scotty79 wrote:
| The fact that they got different numerical results from the
| same data and the difference wasn't explained by the choice
| of statistical tools the applied us very concering.
|
| I interpret it to mean that different researchers applied
| same math to same data and got different results.
| et1337 wrote:
| Their chosen hypothesis doesn't seem any more unprovable than
| the countless other "study says" headlines I see every day,
| even here on HN.
| spullara wrote:
| Most subjects that have the word "science" in the name aren't
| science: social science, computer science, political science,
| etc.
|
| While the real sciences, don't have it: physics, chemistry,
| biology, etc.
| jltsiren wrote:
| That's a tired argument. Those subjects are called the way they
| are, because they were named in a culture that preferred plain
| straightforward names. Computer science could have been
| computing, informatics, or datalogy. Political science could
| have been politology. Sociology is already a thing, but it's
| narrower in scope than social science.
|
| Incidentally, I'm writing this in a place called "Physical
| Sciences Building".
| taylodl wrote:
| Remember the saying figures don't lie but liars figure? I'm not
| saying any of these researchers had any malfeasance, but imagine
| if they did? You can see how easy it is to use "analysis" to say
| anything you want. The key takeaway is don't don't just look at
| the results of the analysis, look at the data and how they
| analyzed it.
| hdjjhhvvhga wrote:
| There is another paper by the same author about a similar
| phenomenon: "Secondary observer effects: idiosyncratic errors in
| small-N secondary data analysis" - it's available from Sci-Hub.
| bluejay2387 wrote:
| I think the timing of articles like this, the recent alarms over
| 'reproducibility' and so on... are a sign that we finally have
| the tools and volume of data to prove that humans are not good
| with data.
| cs702 wrote:
| _> Seventy-three independent research teams used identical cross-
| country survey data to test an established social science
| hypothesis: that more immigration will reduce public support for
| government provision of social policies. Instead of convergence,
| teams' numerical results varied greatly, ranging from large
| negative to large positive effects of immigration on public
| support. The choices made by the research teams in designing
| their statistical tests explain very little of this variation: a
| hidden universe of uncertainty remains._
|
| That's... surprising, but keep in mind that we're talking about
| "soft" social sciences here.
|
| Measuring soft quantities like "public support for government
| provision of social policies in response to increased
| immigration" is _very_ different from measuring hard quantities
| like, say, the location of a rocket in space at a point in time
| using earth as a frame of reference.
|
| I'm looking forward to seeing what Andrew Gelman and his
| colleagues will say about it at
| https://statmodeling.stat.columbia.edu/ -- surely they will want
| to say something about it!
| [deleted]
| photochemsyn wrote:
| Yes, this is a pretty fuzzy 'science' topic - i.e. measuring
| public opinion. Looking at something a bit more quantitative
| would make sense. For example, infectious disease epidemiology
| is kind of mid-way between hard and soft. Looking at Covid
| transmission and outcomes relative to vaccination and testing,
| for example, might fit the bill. There's a good deal of
| uncertainty in the data (lack of reporting, widespread
| differences in prevalence of testing, efficiency of vaccines
| relative to different Covid strains, etc.) but ultimately it
| revolves around a hard science measurement (was person X
| infected with virus Y).
___________________________________________________________________
(page generated 2022-10-17 23:01 UTC)