https://statmodeling.stat.columbia.edu/2022/02/07/the-claimed-effect-size-is-about-a-zillion-times-higher-than-is-plausible/ Skip to primary content Statistical Modeling, Causal Inference, and Social Science Search [ ] [Search] Main menu * Home * Authors * Blogs We Read * Sponsors Post navigation << Thinking Bayesianly about the being-behind-at-halftime effect in basketball Free Bayesian Data Analysis course targeted for Global south >> The claimed effect size is about a zillion times higher than is plausible. Posted on February 7, 2022 9:45 AM by Andrew Joe Hilgard writes: Some years ago, you blogged about a research article by Hasan and colleagues (2013). I had tried to direct your attention to the narrowness of the error bars, which I found suspicious. What I was really trying to say was that the effect size was much, much too big -- by day 3, it is 3.5 standard deviations, or an R^2 of 78%. I've finally managed to publish an article pointing out how implausibly massive that effect size is. I find that a much stronger manipulation yields a large effect size that is still only half as large as the original authors' effect. It seems that there's some kind of serious error in the original research. Hilgard's paper is called "Maximal positive controls: A method for estimating the largest plausible effect size." Also, one of the authors of the earlier paper is Brad "Voodoo" Bushman. I wouldn't trust anything Brad Bushman writes. In that earlier paper, Bushman and his colleagues characterized a three-day study as "long term." So just beyond all the problems with statistics and data errors, we're talking about people who don't even accurately describe their own research claims. Getting back to Hilgard's general point: I find it frustrating when researchers don't think about their effect sizes. This came up in some of our posts years ago on the claim about ovulation and voting (that women's vote preferences change by 20 percentage points during different times of the month) and ovulation and clothing (that women were three times more likely to wear certain colors during certain times of the month). These are (a) ridiculously large effect sizes, (b) even more so given that everything is measured with error, which should cause attenuation of effect size estimates, and (c) very obviously explainable as the result of random chance. But the researchers in question would just not let go. On the plus side, all this discussion motivated a better understanding of experimental design, so future researchers can be helped even if some practitioners of old methods refuse to change their ways. P.S. Hilgard adds: I think that the post underplays what is potentially most scandalous here. It's one thing to publish just-significant results that capitalize on sampling error--that might get someone as far as d = 0.8, p = .040. This seems to be a different beast, though--how does this study find an effect nearly twice the size of "men are taller than women?" How did they find similarly massive effects in their 2012 JESP and 2014 conference presentation? If these results are accurate, they are some of the strongest results in all of social psychology. I think there is something wrong. Yeah, I'm reminded of the painful email exchange I had with a psychology professor who was angry that I criticized at ovulation-and-clothing study. I kept asking him whether he really believed that women were three times more likely to wear red or pink during certain times of the month. He kept not answering. I think he recognized at some level that the effect size was ridiculous, but he didn't care, because the purpose of the study was not to estimate an effect size but rather to demonstrate the existence of an effect. And, from that perspective, a huge effect size estimate was good because it represented super-strong evidence! It seems reasonable--even though it isn't!--to think that p = 0.03 with a huge effect size estimate is better than p = 0.03 with a small effect size estimate. Actually, it's not. The huge effect size estimate is just a signal that the estimate is really noisy, and we're in kangaroo territory. This entry was posted in Miscellaneous Statistics, Zombies by Andrew. Bookmark the permalink. 6 thoughts on "The claimed effect size is about a zillion times higher than is plausible." 1. [268519d0]Jonathan (another one) on February 7, 2022 10:34 AM at 10:34 am said: Your "zillion times" estimate is, I think, off by 2.5473 orders of magnitude,. Reply | 2. [bdcf6d6b]Pierre Dragicevic on February 7, 2022 11:33 AM at 11:33 am said: Joe Hilgard's paper looks interesting but I'm hesitant to pay $31.50 to look at it. Reply | + [28e9]Anonymous on February 7, 2022 11:43 AM at 11:43 am said: Preprint: https://psyarxiv.com/ezkh4/ Reply | 3. [08445007]Kevin S Van Horn on February 7, 2022 10:51 PM at 10:51 pm said: I see that you're following your own advice about excessively precise numbers. For any quantity measured in zillions, it is pointless to give more than 0 significant digits. Reply | 4. [6c0517d8]Martin S on February 11, 2022 5:45 PM at 5:45 pm said: I don't understand the point about a large effect size definitely meaning there is noise. What if the null hypothesis were not a null/zero effect but a large effect? Please help me. I wanna believe, but can't. Reply | + [d582]Andrew on February 11, 2022 6:06 PM at 6:06 pm said: Martin: See the first link in the above post. As Hilgard explains, "Some reported effect sizes are too big for the hypothesized process." When we say that a claimed effect size is implausibly large, this statement of implausibility is in reference to other things we know about the world. For example, a few years ago we discussed a paper that appeared to find that women's support for Barack Obama was 20 percentage points higher during certain times of the month. This is not plausible given what we know about political polarization: only a very small percentage of people change their minds on who they are supporting for president, much less than 20%. Another example was the claim that beautiful parents were 8 percentage points more likely to have girls, compared to non-beautiful parents. This is implausible given that the sex ratio has been studied over and over, and never has there been an effect of this size found in any population. On the other hand, these estimated effect sizes are trivially easy to explain as random noise. Reply | Leave a Reply Cancel reply Your email address will not be published. [ ] [ ] [ ] [ ] [ ] [ ] [ ] Comment [ ] Name [ ] Email [ ] Website [ ] [ ] [Post Comment] [ ] [ ] [ ] [ ] [ ] [ ] [ ] D[ ] * Art * Bayesian Statistics * Causal Inference * Decision Theory * Economics * Jobs * Literature * Miscellaneous Science * Miscellaneous Statistics * Multilevel Modeling * Political Science * Public Health * Sociology * Sports * Stan * Statistical computing * Statistical graphics * Teaching * Zombies 1. Andrew on Alexey Guzey on sleep: Put this guy on Rogan and give him a Ted talk.February 25, 2022 4:58 PM Thanks! 2. Billy Buchanan on Alexey Guzey on sleep: Put this guy on Rogan and give him a Ted talk.February 25, 2022 3:08 PM Alhola, P., & Polo-Kantola, P. (2007). Sleep deprivation: Impact on cognitive performance. Neuropsychiatric Disease and Treatment, 3(5), 553-567. https://pubmed.ncbi.nlm.nih.gov/19300585/ Moore,... 3. Joshua on Grover Norquist, Anne-Marie Slaughter, and a rabbi walk into a bar . . .February 25, 2022 2:38 PM Thanks Sameera - > He makes good points too. I also feel that sometiems he does make good points. Unlike... 4. Joshua on "Masks block some fraction of viral particles from dispersing from those who are infected and from infecting those who are susceptible . . . studies that did not find masks to be effective had limited statistical power and therefore do not imply that masks are ineffective . . . The studies that did not find statistically significant effects prove only that masks cannot offer protection if they are not worn."February 25, 2022 2:26 PM Phil - > How people feel about Covid precautions depends strongly on how much they care about other people. Ever... 5. Giovanna Fernandes on Some Bayesian Methods for Causal Inference (my remote talk next Monday at the University of Wisconsin Population Health Sciences Seminar)February 25, 2022 1:09 PM Will this be available for the general public online? Interested in access 6. Matt Skaggs on Weakliem on public opinion and politicsFebruary 25, 2022 12:49 PM The problem with any set of numbers on the issue of abortion is that a large cohort on one side... 7. Sameera Daniels on Grover Norquist, Anne-Marie Slaughter, and a rabbi walk into a bar . . .February 25, 2022 12:34 PM Hi Joshua, I am aware that Prasad blocks individuals on Twitter. I do find that odd, given that he encourages... 8. Ben on Hierarchical model golf putting success!February 25, 2022 12:14 PM Wow that's a great video. 9. Phil on "Masks block some fraction of viral particles from dispersing from those who are infected and from infecting those who are susceptible . . . studies that did not find masks to be effective had limited statistical power and therefore do not imply that masks are ineffective . . . The studies that did not find statistically significant effects prove only that masks cannot offer protection if they are not worn."February 25, 2022 11:36 AM Kevin, How people feel about Covid precautions depends strongly on how much they care about other people. Ever since the... 10. Anonny on Weakliem on public opinion and politicsFebruary 25, 2022 10:55 AM I read the one on institutional trust, and I don't think it's safe to just take the numbers at face... 11. Andrew on "Efficacy of Ivermectin Treatment on Disease Progression Among Adults With Mild to Moderate COVID-19 and Comorbidities The I-TECH Randomized Clinical Trial"February 25, 2022 10:48 AM Sander: I agree with you completely on points 4 and 5. 12. Andrew on Testing fractional doses of COVID-19 vaccinesFebruary 25, 2022 10:44 AM Unanon: Not everyone who contributes to these discussions is a native English speaker, and that's ok! 13. Unanon on Testing fractional doses of COVID-19 vaccinesFebruary 25, 2022 10:24 AM Anonyd00d, it's quite clear to people with native English-comprehension abilities that the authors are clearly not claiming COVID-19 vaccines will... 14. Matt C. on Hierarchical model golf putting success!February 25, 2022 9:05 AM Yes, the misses from <1 ft will mostly be guys who went to hastily tap in a putt (after missing... 15. Andrew on Hierarchical model golf putting success!February 25, 2022 7:32 AM Blogger Vance. 16. Phil on "Masks block some fraction of viral particles from dispersing from those who are infected and from infecting those who are susceptible . . . studies that did not find masks to be effective had limited statistical power and therefore do not imply that masks are ineffective . . . The studies that did not find statistically significant effects prove only that masks cannot offer protection if they are not worn."February 25, 2022 1:23 AM oncodoc, you say "as the article points out evidence to support masking is hard to find." What article are you... 17. Joshua on Grover Norquist, Anne-Marie Slaughter, and a rabbi walk into a bar . . .February 24, 2022 11:59 PM Hi Sameera - Have you seen Vinay interview people he disagrees strongly with? Seems to me he (always?) likes interviewing... 18. Joshua on "Efficacy of Ivermectin Treatment on Disease Progression Among Adults With Mild to Moderate COVID-19 and Comorbidities The I-TECH Randomized Clinical Trial"February 24, 2022 11:22 PM Greg - I was wondering if it might be related to the worm/ corticoseroids/ivermectin interaction. I was surprised no one (I've... 19. Greg Tucker-Kellogg on "Efficacy of Ivermectin Treatment on Disease Progression Among Adults With Mild to Moderate COVID-19 and Comorbidities The I-TECH Randomized Clinical Trial"February 24, 2022 9:46 PM Joshua, I'd like to add to my comments about secondary outcomes. The observation that 4/10 deaths in the control group... 20. a on "Masks block some fraction of viral particles from dispersing from those who are infected and from infecting those who are susceptible . . . studies that did not find masks to be effective had limited statistical power and therefore do not imply that masks are ineffective . . . The studies that did not find statistically significant effects prove only that masks cannot offer protection if they are not worn."February 24, 2022 9:41 PM Good point about electrostatics and N95 masks. My understanding of the study we are talking about is that it makes... 21. rsm on Hierarchical model golf putting success!February 24, 2022 8:37 PM Do you really want to be some kind of statistical Bagger Vance, at Mar-a-lago? 22. Josbua on Testing fractional doses of COVID-19 vaccinesFebruary 24, 2022 8:06 PM Navigator - > BTW, there is a very slim evidence that vaccination status would have anything to do with 'long... 23. Sameera Daniels on Grover Norquist, Anne-Marie Slaughter, and a rabbi walk into a bar . . .February 24, 2022 7:55 PM I don't need to pay that amount to talk to Anne-Marie Slaughter. With all these planned events, we've underappreciated 'serendipity'... 24. Andrew on Grover Norquist, Anne-Marie Slaughter, and a rabbi walk into a bar . . .February 24, 2022 7:49 PM It's also possible that Anne-Marie Slaughter wants some foreign policy tips, or that the rabbi wants to quiz me on... 25. Anoneuoid on Testing fractional doses of COVID-19 vaccines February 24, 2022 7:19 PM who exactly in this thread is "continuing to act like the vaccines are 95% effective at preventing symptomatic illness"? [...]... 26. Sameera on "Masks block some fraction of viral particles from dispersing from those who are infected and from infecting those who are susceptible . . . studies that did not find masks to be effective had limited statistical power and therefore do not imply that masks are ineffective . . . The studies that did not find statistically significant effects prove only that masks cannot offer protection if they are not worn."February 24, 2022 7:16 PM Vinay Prasad has been discussing the masking situation for several months. Here is a YouTube that I found intriguing. https: //www.youtube.com/watch?v=tmQisBCZl_A 27. Mikhail Shubin on Pen and notebook recommendations?February 24, 2022 7:01 PM I just usually use pen and paper stacks of paper I borrow from conference hotels 28. Andrew on Grover Norquist, Anne-Marie Slaughter, and a rabbi walk into a bar . . .February 24, 2022 6:11 PM I dunno, maybe Grover thought I'd have something interesting to say? 29. Sam on Grover Norquist, Anne-Marie Slaughter, and a rabbi walk into a bar . . .February 24, 2022 6:07 PM Why do you think you were invited? 30. Sander Greenland on "Efficacy of Ivermectin Treatment on Disease Progression Among Adults With Mild to Moderate COVID-19 and Comorbidities The I-TECH Randomized Clinical Trial"February 24, 2022 6:07 PM Andrew: OK, thanks for clarifying your positions... 2. We disagree about the relevance of the larger inclusion (smaller exclusion) meta-analysis:... 31. Unanon on Testing fractional doses of COVID-19 vaccinesFebruary 24, 2022 5:34 PM Anonyd00d, assuming you are trying to remain on-topic, who exactly in this thread is "continuing to act like the vaccines... 32. Navigator on "Masks block some fraction of viral particles from dispersing from those who are infected and from infecting those who are susceptible . . . studies that did not find masks to be effective had limited statistical power and therefore do not imply that masks are ineffective . . . The studies that did not find statistically significant effects prove only that masks cannot offer protection if they are not worn."February 24, 2022 4:45 PM Exactly, Electrostatics is what makes it very effective. I'd also add fit. There are so many different face shapes, but... 33. Andrew on "Efficacy of Ivermectin Treatment on Disease Progression Among Adults With Mild to Moderate COVID-19 and Comorbidities The I-TECH Randomized Clinical Trial"February 24, 2022 4:25 PM Sander: 1. Thanks for pointing out the cut-and-paste error in my comment; I fixed it. 2. When I said you... 34. Daniel Lakeland on Testing fractional doses of COVID-19 vaccines February 24, 2022 4:05 PM Sounds like a histamine reaction maybe. Probably something other than the active ingredient. Someone should be doing trials of the... 35. Anoneuoid on Testing fractional doses of COVID-19 vaccines February 24, 2022 4:05 PM Everyone is having the same anecdotes, as also indicated by more cases than ever. It is clear there are many... 36. Sander Greenland on "Efficacy of Ivermectin Treatment on Disease Progression Among Adults With Mild to Moderate COVID-19 and Comorbidities The I-TECH Randomized Clinical Trial"February 24, 2022 3:55 PM Andrew: First, your post at its end shows on my screen as reproducing what I wrote as if you wrote... 37. Carlos Ungil on "Efficacy of Ivermectin Treatment on Disease Progression Among Adults With Mild to Moderate COVID-19 and Comorbidities The I-TECH Randomized Clinical Trial"February 24, 2022 3:26 PM > You also wrote: "As far as I know there isn't a collection of successful and unsuccessful trials for molnupiravir.":... 38. Kevin S Van Horn on "Masks block some fraction of viral particles from dispersing from those who are infected and from infecting those who are susceptible . . . studies that did not find masks to be effective had limited statistical power and therefore do not imply that masks are ineffective . . . The studies that did not find statistically significant effects prove only that masks cannot offer protection if they are not worn."February 24, 2022 3:06 PM "I spend some time every day hanging out with my 81-year-old mother" That's a good reason to be scrupulous about... 39. Andrew on "Efficacy of Ivermectin Treatment on Disease Progression Among Adults With Mild to Moderate COVID-19 and Comorbidities The I-TECH Randomized Clinical Trial"February 24, 2022 3:00 PM Sander: You write: As Marinos whittles the study pool down, the summary estimate stays in the 60-70% reduction range for... 40. Unanon on Testing fractional doses of COVID-19 vaccinesFebruary 24, 2022 2:57 PM "This is the second time you've shared this anecdote. Both times you appear to have not understood the purpose of... 41. Sander Greenland on "Efficacy of Ivermectin Treatment on Disease Progression Among Adults With Mild to Moderate COVID-19 and Comorbidities The I-TECH Randomized Clinical Trial"February 24, 2022 2:50 PM Thanks Carlos for the NEJM link. Notable perhaps is how (again) the more severe outcome has a far larger estimated... 42. Joshua on "Efficacy of Ivermectin Treatment on Disease Progression Among Adults With Mild to Moderate COVID-19 and Comorbidities The I-TECH Randomized Clinical Trial"February 24, 2022 2:22 PM As a fan of irony, I do want again to point out an amusing overlap of the conspiracy-hued reasoning behind... 43. Navigator on Testing fractional doses of COVID-19 vaccines February 24, 2022 1:57 PM Vaccines work but not in a way many people think. Immunity is not one thing. There is innate and acquired... 44. JimV on "Masks block some fraction of viral particles from dispersing from those who are infected and from infecting those who are susceptible . . . studies that did not find masks to be effective had limited statistical power and therefore do not imply that masks are ineffective . . . The studies that did not find statistically significant effects prove only that masks cannot offer protection if they are not worn."February 24, 2022 1:43 PM It is my understanding that viruses are not free-floating in air but are carried in the water droplets which we... 45. oncodoc on "Masks block some fraction of viral particles from dispersing from those who are infected and from infecting those who are susceptible . . . studies that did not find masks to be effective had limited statistical power and therefore do not imply that masks are ineffective . . . The studies that did not find statistically significant effects prove only that masks cannot offer protection if they are not worn."February 24, 2022 1:39 PM I'm 100% compliant with the rules about masking currently in force, and I understand the rationale. However, as the article... 46. Daniel Lakeland on "Masks block some fraction of viral particles from dispersing from those who are infected and from infecting those who are susceptible . . . studies that did not find masks to be effective had limited statistical power and therefore do not imply that masks are ineffective . . . The studies that did not find statistically significant effects prove only that masks cannot offer protection if they are not worn."February 24, 2022 1:35 PM Nah, the mechanism of action for N95s is electrostatic attraction. For non-electrostatic particles the mechanism is tortuosity. Pebble sized rocks... 47. Daniel Lakeland on Testing fractional doses of COVID-19 vaccines February 24, 2022 1:27 PM I don't think there's anyone who knows any biology at all that thinks we'll get away with a single vaccine... 48. JFA on "Masks block some fraction of viral particles from dispersing from those who are infected and from infecting those who are susceptible . . . studies that did not find masks to be effective had limited statistical power and therefore do not imply that masks are ineffective . . . The studies that did not find statistically significant effects prove only that masks cannot offer protection if they are not worn."February 24, 2022 1:08 PM I would note that the figure reproduced in this post does not use any post pandemic studies (they are all... 49. Anoneuoid on Testing fractional doses of COVID-19 vaccines February 24, 2022 1:03 PM I've said from the beginning that the vaccines will protect against viremia, which essentially means severe disease. They will do... 50. Carlos Ungil on "Efficacy of Ivermectin Treatment on Disease Progression Among Adults With Mild to Moderate COVID-19 and Comorbidities The I-TECH Randomized Clinical Trial"February 24, 2022 12:56 PM > What is available indicates that molnupiravir has no more benefit (30% risk reduction when properly timed) than does ivermectin... Proudly powered by WordPress