https://statmodeling.stat.columbia.edu/2022/10/05/not-frequentist-enough/ Skip to primary content Statistical Modeling, Causal Inference, and Social Science Search [ ] [Search] Main menu * Home * Authors * Blogs We Read * Sponsors Post navigation Rich guys and their dumb graphs: The visual equivalents of "Dow 36,000" What Nested R-hat teaches us about the classical R-hat Not frequentist enough. Posted on October 5, 2022 9:58 AM by Andrew I think that many mistakes in applied statistics could be avoided if people were to think in a more frequentist way. Look at it this way: In the usual way of thinking, you apply a statistical procedure to the data, and if the result reaches some statistical-significance threshold, and you get similar results from a robustness study, changing some things around, then you've made a discovery. In the frequentist way of thinking, you consider your entire procedure (all the steps above) as a single unit, and you consider what would happen if you apply this procedure to a long series of similar problems. The first thing to recognize is that the frequentist way of thinking requires extra effort: you need to define this potential series of similar problems and then either do some mathematical analysis or, more likely, set up a simulation on the computer. The second thing to recognize is that, just because a statistical method is defined in a "classical" or hypothesis-testing framework, that doesn't make it a "frequentist" method. For a method to be frequentist, it needs to defined relative to some frequency calibration. A p-value or a confidence interval, by itself, is not frequentist; for it to be frequentist there needs to be some model of what would be done in a family of replications of the procedure. This is the point that Loken and I make in section 1.2 of our forking paths paper. In the usual way of teaching statistics, the extra effort required by the frequentist approach is not clear, for two reasons. First, textbooks present the general theory in the context of simple examples such as linear models with no selection, where there are simple analytic solutions. Second, textbook examples of statistical theory typically start with an assumed probability model for the data, in which case most of the hard work has already been done. The model is just there, postulated; it doesn't look like a set of "assumptions" at all. It's the camel that is the likelihood (although, strictly speaking, the likelihood is not the data model; additional assumptions are required to go from an (unnormalized) likelihood function to get a generative model for the data). An example To demonstrate this point, I'll use an example from a recent article, Criticism as asynchronous collaboration: An example from social science research, where I discussed a published data analysis that claimed to show that "politicians winning a close election live 5-10 years longer than candidates who lose," with this claim being based on a point estimate from a few hundred elections: the estimate was statistically significantly different from zero and similar estimates were produced in a robustness study in which various aspects of the model were tweaked. The published analysis was done using what I describe above as "the usual way of thinking." Now let's consider the frequentist approach. We have to make some assumptions. Suppose you start with the assumption that losing an election has the effect of increasing your lifespan by X years, where X has some value between -1 and 1. (From an epidemiological point of view, an effect of 1 year is large, really on the very high end of what could be expected as an average treatment effect of something as indirect as winning or losing an election.) From there you can work out what might happen from a few hundred elections, and you'll see that any estimate will be super noisy, to the extent that if you fit a model and select on statistical significance, you'll get an estimated effect that's much higher than the real effect (a large type M error, as we say). You'll also see that, if you want to get a large effect (large effects are exciting, right!) then you'll want the standard error of your estimate to be larger, and you can get this by the simple expedient of predicting future length of life without including current age as a predictor. For more discussion of all these issues, see section 4 of the linked article. My point here is that whatever analysis we do, there is a benefit to thinking about it from a frequentist perspective--what would things look like if the procedure were replied repeatedly to many datasets?--rather than to fixate on the results of the analysis as applied to the data at hand. This entry was posted in Bayesian Statistics, Decision Theory, Miscellaneous Statistics, Political Science, Zombies by Andrew. Bookmark the permalink. 4 thoughts on "Not frequentist enough." 1. [4bab043b]psyoskeptic on October 5, 2022 12:30 PM at 12:30 pm said: yes Reply | 2. [6ac1df32]Ethan on October 5, 2022 12:47 PM at 12:47 pm said: > what would things look like if the procedure were replied repeatedly to many datasets? And, with modern computation power, you might as well create a full simulation as well! Reply | 3. []jbayes on October 6, 2022 9:14 AM at 9:14 am said: If you do the best you can on each individual "entire procedure ... as a single unit" then you can't help but do well in "a long series of similar problems". The converse, however, is not true. Doing well in a long series doesn't imply you'll do the best you can in on each individual procedure. A better name for this might be "The Frequentist trap". Reply | 4. [4b06543d]jbayes on October 6, 2022 9:14 AM at 9:14 am said: If you do the best you can on each individual "entire procedure ... as a single unit" then you can't help but do well in "a long series of similar problems". The converse, however, is not true. Doing well in a long series doesn't imply you'll do the best you can in on each individual procedure. A better name for this might be "The Frequentist trap". Reply | Leave a Reply Cancel reply Your email address will not be published. Required fields are marked * [ ] [ ] [ ] [ ] [ ] [ ] [ ] Comment * [ ] Name [ ] Email [ ] Website [ ] [Post Comment] [ ] [ ] [ ] [ ] [ ] [ ] [ ] D[ ] * Art * Bayesian Statistics * Causal Inference * Decision Theory * Economics * Jobs * Literature * Miscellaneous Science * Miscellaneous Statistics * Multilevel Modeling * Papers * Political Science * Public Health * Sociology * Sports * Stan * Statistical computing * Statistical graphics * Teaching * Zombies 1. billb on Mira magic--a probabilistic card trickOctober 7, 2022 6:26 PM I had a nice morning programming in R a simulation for this. I wanted to get a sense of what... 2. billb on Mira magic--a probabilistic card trickOctober 7, 2022 5:42 PM I see, in python you're 0 indexed... 3. billb on Mira magic--a probabilistic card trickOctober 7, 2022 5:34 PM shouldn't this condition in the while loop be: while pos + deck [pos] < = len(deck) ?? 4. samuel on Mira magic--a probabilistic card trickOctober 7, 2022 4:31 PM "Next, each kid chooses a starting point from among the first five cards and stands on the card they have... 5. somebody on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 4:30 PM You're talking about a microchip with onboard accelerometers for tracking finger position, which then processes that accelerometer data to read... 6. somebody on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 4:16 PM I assume putting a camera in the players hotel room to watch them practice before the match would be cheating.... 7. Carlos Ungil on Mira magic--a probabilistic card trickOctober 7, 2022 3:41 PM That's a nice trick. Having multiple people doing it in parallel gives it away and spoils the mystery - especially... 8. Anoneuoid on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 2:49 PM Carlsen had never played this line before, even online, so it would be pretty amazing if Niemann had prepared it.... 9. Charles Margossian on What Nested R-hat teaches us about the classical R-hatOctober 7, 2022 2:41 PM hmmm... I'd be intrigued to work out in details what makes the nonparameteric case different and whether it would make... 10. Anoneuoid on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 2:38 PM @Somebody I assume putting a camera in the players hotel room to watch them practice before the match would be... 11. Andrew on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 2:01 PM Dyson vs. Roomba. 12. different anon on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 1:57 PM "But it did have a novelty on move 15 -- a move never before seen in recorded play -- so... 13. Phil on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 1:32 PM Agreed, statistical methods of catching cheaters by comparing to computer play are never going to be infallible. They would work... 14. Phil on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 1:21 PM It's not just microwaves! Lots of appliances are getting smart now. Watch out for that vacuum cleaner! It's not as... 15. Phil on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 1:19 PM David, Yes, Chess.com is being very careful not to say they think he cheated. But that's very different from your... 16. Phil on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 1:04 PM Hahaha you're criticizing the post for not citing a specific person who you think should be cited? Get real. I'm... 17. Phil on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 1:02 PM Thanks, Andrew! You did a pretty good job speaking for me but I'll speak for me too. 1 is pedantic... 18. Peter Dorman on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 12:43 PM Well, I guess it was just a matter of time before the Niemann cheating maybe-scandal arrived at this site. It's... 19. somebody on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 12:42 PM The easiest way I'm thinking to overcome that obstacle is get access to what the opponent is practicing before the... 20. Andrew on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 12:19 PM Anon, I can't speak for Phil, but: 1. OK, Carlson didn't say that particular phrase, he tweeted someone else saying... 21. Anonymous on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 12:07 PM This is not the only weakness in the post. Here are some others that stood out to me (I'm not... 22. Jackson on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 11:55 AM 'But in the end, the underlying issue is that chess is now a "solved problem".' This is totally incorrect. Checkers... 23. David in Tokyo on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 11:15 AM Here's Niemann's first game from the US Open. https:// www.youtube.com/watch?v=MtDvnFwAD4s&ab_channel= agadmator%27sChessChannel 24. Andrew on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 10:59 AM Nah, I get destroyed by the people who think three steps ahead. 25. Andrew on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 10:58 AM Matt: Google *historical incidence of otb cheating by electronic signal* and you'll find some things. 26. Joshua on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 10:57 AM You're always thinking two steps ahead, Andrew. You should consider playing chess. 27. Joshua on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 10:56 AM Anoneuoid - > I have noticed that I regularly do "impossible" things according to some people who use NHST* reasoning... 28. Matt Skaggs on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 10:54 AM It really would help if someone could point to any historical incidence of OTB cheating by electronic signal and how... 29. paul alper on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 10:11 AM Maybe someone has already posted this explanation: https:// www.huffpost.com/entry/ chess-grandmaster-butt-scan_n_63400437e4b08e0e6077ad0b Anyone who monitors a large class taking a statistics exam is now... 30. Andrew on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 9:57 AM Joshua: That would be part of security, that the arbiter would check any microwave ovens in the hall to make... 31. Joahua on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 9:38 AM Andrew - That's an absurd comment that's obviously just wrong. Some microwaves have a setting to silence the bell. So... 32. David in Tokyo on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 9:18 AM You misunderstand me: what I was saying there was that my best guess is that Niemann is a talented chess... 33. Barney on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 6:03 AM "Whatever. I remain of the opinion that it's unlikely that Niemann cheating over the board has a lot to do... 34. Florian Wickelmaier on Numb3d by Numbers--a must see kid's science exhibition about measurement, counting, and statsOctober 7, 2022 6:00 AM Here is a quick attempt at the horse-race simulation: ``` race <- setNames(rep(0, 11), 2:12) while(all(race < 10)) { winner... 35. Barney on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 5:49 AM The measure of improvement that is being used relates to ELO increases. A player's ELO changes as a function of... 36. Zhou Fang on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 5:02 AM > The chess.com folks/paper seem pretty clear that they don't think that he cheated in OTB chess, No, they are... 37. Andrew on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 3:35 AM Phil: The main message of your post seems to be that anybody can cheat in OTB by just sneaking a... 38. David in Tokyo on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 2:45 AM "I suppose this depends on what you mean by "inaccurate."" Yes. The last time I checked, rating calculations limit delta... 39. Anonymous on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 2:31 AM A minor correction: The original post says "In both of those cases, Carlsen and Niemann were playing "over the board"... 40. David in Tokyo on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 1:46 AM Chess is a really intense game. In the very last tournament I played, there was (at least but probably not... 41. David in Tokyo on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 1:17 AM What Phil said. OTB will remain fine. One thing that crossed my mind, though, is should people who have cheated... 42. Phil on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 1:16 AM "So a big component of that improvement could have been an inaccurate rating."... I suppose this depends on what you... 43. Phil on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 1:01 AM There are some chess openings for which known lines go on forever, by which I mean fifteen moves or more.... 44. Roger Schlafly on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 1:00 AM Niemann is currently playing in the US Chess Open in St. Louis, where strong anti-cheating precautions have been taken. Metal... 45. Anoneuoid on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 12:51 AM The easiest way I'm thinking to overcome that obstacle is get access to what the opponent is practicing before the... 46. Anonymous on (Now that faculty aren't coming into the office anymore) Will universities ever recover?October 7, 2022 12:35 AM Govt of India is trying to replace all classroom teaching with online NPTEL MOOCs courses. This benefits many who have... 47. Phil on Chess cheating: how to detect it (other than catching someone with a shoe phone)October 7, 2022 12:26 AM In OTB chess I think it should be possible to prevent cheating, at least in high-level tournaments for which it's... 48. Alex Vasilescu on (Now that faculty aren't coming into the office anymore) Will universities ever recover?October 7, 2022 12:13 AM I found Facebook to be akin to a virtual water cooler. Sometimes colleagues share to the feed tech articles with... 49. Yuling Yao on Bayesian inference for discrete parameters and Bayesian inference for continuous parameters: Are these two completely different forms of inference?October 7, 2022 12:07 AM Carlos, actually I am under the impression that it is an important character for an (counter)example/paradox to be goofy, or... 50. Yuling Yao on (Now that faculty aren't coming into the office anymore) Will universities ever recover?October 6, 2022 11:57 PM My limited impression is that, it is not a coincidence that corporate America has learned how to use free/accessible meals/ celery-sticks... Proudly powered by WordPress