Post ASrsbw3g7FA8PI0fYW by lakens@mastodon.social
(DIR) More posts by lakens@mastodon.social
(DIR) Post #ASrsbqD1rZ4qGYL3IW by raoulvanoosten@ecoevo.social
2023-02-15T09:49:51Z
0 likes, 0 repeats
Greenland et al. (2016, https://doi.org/10.1007/s10654-016-0149-3 ) suggested a more refined goal of statistics than testing study hypotheses is the evaluation of the (un)certainty of effect sizes.I agree. The majority of studies are exploratory, hence testing hypotheses does not make sense.#science #statistics #papers
(DIR) Post #ASrsbqjHvcshsbQoa0 by lakens@mastodon.social
2023-02-15T10:29:05Z
0 likes, 0 repeats
@raoulvanoosten Don't you think this conflates testing (distinguishing signal from noise) and testing hypotheses (confirming or falsifying theoretical predictions)? Even if you do not want to do the second, you often want to do the first, no?
(DIR) Post #ASrsbrQXKpTa2jfLzU by raoulvanoosten@ecoevo.social
2023-02-15T11:45:19Z
0 likes, 0 repeats
@lakens do you mean estimation of ES (un)certainty is enough both to distinguish signal from noise and to test hypotheses?
(DIR) Post #ASrsbrr7jyk9NC6aQq by lakens@mastodon.social
2023-02-15T11:48:26Z
0 likes, 0 repeats
@raoulvanoosten That ius exactly the discussion - are you happy just estimating an effect size, and discussing it, even if that effect size can be noise? If so, then you do not need to test. But in practice, this is not what I see in papers - even in estimation papers. People often declare something 'an effect' -not just 'an estimate'
(DIR) Post #ASrsbsUpMMVDMKgIJk by raoulvanoosten@ecoevo.social
2023-02-15T12:48:47Z
0 likes, 0 repeats
@lakens I'm not sure yet. I think significance testing (with p < .05) is a bad idea, while a priori determination of when claims will be accepted or refuted is essential. I am undecided whether ES and CIs are enough (I have not come across other frequentist methods yet).
(DIR) Post #ASrsbsyFay2QpaRnBA by lakens@mastodon.social
2023-02-15T12:51:15Z
0 likes, 0 repeats
@raoulvanoosten I think it is currently fashionable to think p<.05 is bad, while almost everyone thinks it is a useful tool that, as all statistics, is often used without sufficient training. The difference is important. I have never seen anyone provide a coherent view on making claims without error control
(DIR) Post #ASrsbtYPQWxgdjMfXU by raoulvanoosten@ecoevo.social
2023-02-16T14:27:46Z
0 likes, 0 repeats
@lakens it is definitely fashionable to criticize NHST and the default p < .05. I think the good thing about that is that researchers shouldn't use these defaults without thinking about them. Which is mostly about education, like you say.
(DIR) Post #ASrsbtxvtdNVutJ3K4 by lakens@mastodon.social
2023-02-16T16:21:16Z
0 likes, 0 repeats
@raoulvanoosten I agree. And I think it is good to show how to do it better in practice. This step does not happen a lot - that was the whole point of the 'moving beyond p < 0.05' special issue - and even in that special issue, most solutions STILL recommended p-values, but proposed to add some additional statistic.
(DIR) Post #ASrsbuSQ4HlTRRZOqG by raoulvanoosten@ecoevo.social
2023-02-16T16:48:04Z
0 likes, 0 repeats
@lakens but the editors of that special issue suggest "don't use statistical significance". That's also my current standpoint: p-values are fine as a continuous metric, and more education is needed so people use them correctly.Hypothesis testing needs more, and I'm unsure yet what.
(DIR) Post #ASrsbuwuEw9QxzpkMS by lakens@mastodon.social
2023-02-16T17:01:15Z
0 likes, 0 repeats
@raoulvanoosten The editors of that special issue are, regrettably, incompetent biased unscientific individuals who, with their very bad editorial, led to a taskforce that had to correct their mistakes https://doi.org/10.1080/09332480.2021.2003631 so that people like you would not be misguided by them. So, I am very sorry if I just ignore that editorial altogether, and listen to more competent people.
(DIR) Post #ASrsbverbVJTAKOqsS by raoulvanoosten@ecoevo.social
2023-02-16T17:43:59Z
0 likes, 0 repeats
@lakens subtle :P. I did not know this. I'll check out that paper. Thanks.
(DIR) Post #ASrsbw3g7FA8PI0fYW by lakens@mastodon.social
2023-02-16T17:45:31Z
0 likes, 0 repeats
@raoulvanoosten The emotion is strong. I am writing a blog about this (which I rarely do anymore). They also lie that most of the articles in the special issue agree with their view. Not at all true. So unscientific. So misleading.
(DIR) Post #ASrsbwWOOU8BqLRbJQ by raoulvanoosten@ecoevo.social
2023-02-17T08:14:35Z
0 likes, 0 repeats
@lakens aw man, that sucks. Thanks for the Task Force paper. It seems I should check out all papers in the special issue myself, too.
(DIR) Post #ASrsbx1wVBMtQCCnUO by lakens@mastodon.social
2023-02-17T08:40:52Z
0 likes, 0 repeats
@raoulvanoosten I added a short review in our revised version of https://psyarxiv.com/af9by/. The screenshots are the relevant paragraphs. I wanted to write a blog about it to discuss this in more detail (we are very brief in the paper).
(DIR) Post #ASrsbxaKRKsF8qIG5Q by raoulvanoosten@ecoevo.social
2023-02-17T09:41:16Z
0 likes, 0 repeats
@lakens so using interval hypothesis tests (and meaningful effect sizes)? When I started reading about the NHST-issue about a year ago (with papers like Meehl, 1967), that's what I thought but I couldn't find concrete examples and ended up with the "abandon significance" idea. Good to hear there is a body of work that supports the use of interval hypotheses (I was reading your book, which seems like a good continuation).
(DIR) Post #ASrsbyCc8zUz3aCplI by david_colquhoun@mstdn.social
2023-02-17T10:47:43Z
0 likes, 0 repeats
@raoulvanoosten @lakens The problem with confidence intervals is that they are random variables, as are p values. That means that when you calculate CI for your experiment, it is not right to say there's a 95% chance that the true value is included in the CI. A repeat of the experiment will give different CI.
(DIR) Post #ASrsbygkKxbMZ2ItjE by lakens@mastodon.social
2023-02-17T08:42:14Z
0 likes, 0 repeats
@raoulvanoosten It would be interesting to hear your thoughts if you manage to read all 43 papers in the special issue :) I might so far be the only person who did so 😅 It is a long read (not all papers are equally good). But it is interesting. Yes, a review would be good! Could be a paper in itself, you are right!
(DIR) Post #ASrsbysRdSxX9JmExk by raoulvanoosten@ecoevo.social
2023-02-17T11:48:53Z
0 likes, 0 repeats
@david_colquhoun so that's what Greenland et al (2016, # 202301051039 greenland2016) meant. So are they still useful?
(DIR) Post #ASrsbzRBYIkSt41z72 by david_colquhoun@mstdn.social
2023-02-17T11:58:56Z
0 likes, 0 repeats
@raoulvanoosten Are CI still useful? The problem is that people tie themselves in knots when trying to define how they should be interpreted, just as they do with p values. That's why I love the Wagenmakers quotation. I've advocated supplementing the p value and CI (rather than abandoning them) with a likelihood ratio or some measure of false positive risk
(DIR) Post #ASrsc04tAgVWsCbgzw by raoulvanoosten@ecoevo.social
2023-02-19T07:32:46Z
0 likes, 0 repeats
@david_colquhoun @lakens I think indeed frequentist interference tries to answer the wrong question ("what is the probability of finding these data under the tested hypothesis?" rather than "what is the probability my tested hypothesis is true?"). Your false positive risk ( https://doi.org/10.1098/rsos.171085 ) is much closer to that question.
(DIR) Post #ASrsc0cv89jIZkWs2i by lakens@mastodon.social
2023-02-19T07:41:13Z
0 likes, 0 repeats
@raoulvanoosten @david_colquhoun just a reminder that we can never answer the question 'what is the probability my hypothesis is true' in science. Frequenties is not answering the wrong question. It is answering the right, and only, question.
(DIR) Post #ASrsc1AF8GNuF67Tyy by david_colquhoun@mstdn.social
2023-02-20T10:31:17Z
0 likes, 1 repeats
@lakens @raoulvanoosten The problem with p values is that they confuse two quite different quantities.The probability that you have 4 legs given that you are a cow is high.The probability that you are a cow, given that you have 4 legs is low.