https://statmodeling.stat.columbia.edu/2022/09/12/whats-the-difference-between-derek-jeter-and-preregistration/ Skip to primary content Statistical Modeling, Causal Inference, and Social Science Search [ ] [Search] Main menu * Home * Authors * Blogs We Read * Sponsors Post navigation "Merchants of doubt" operating within organized science Geoff Dyer Kazuo Ishiguro Owen Sheers David Leavitt Veronica Geng What's the difference between Derek Jeter and preregistration? Posted on September 12, 2022 9:34 AM by Andrew There are probably lots of clever answers to this one, but I'll go with: One of them was hyped in the media as a clean-cut fresh face that would restore fan confidence in a tired, scandal-plagued entertainment cartel--and the other is a retired baseball player. Let me put it another way. Derek Jeter had three salient attributes: 1. He was an excellent baseball player, rated by one source at the time of his retirement as the 58th best position player of all time. 2. He was famously overrated. 3. He was a symbol of integrity. The challenge is to hold 1 and 2 together in your mind. I was thinking about this after Palko pointed me to a recent article by Rose McDermott that begins: Pre-registration has become an increasingly popular proposal to address concerns regarding questionable research practices. Yet preregistration does not necessarily solve these problems. It also causes additional problems, including raising costs for more junior and less resourced scholars. In addition, pre-registration restricts creativity and diminishes the broader scientific enterprise. In this way, pre-registration neither solves the problems it is intended to address, nor does it come without costs. Pre-registration is neither necessary nor sufficient for producing novel or ethical work. In short, pre-registration represents a form of virtue signaling that is more performative than actual. I think this is like saying, "Derek Jeter is no Cal Ripken, he's overrated, gets too much credit for being in the right place at the right time, he made the Yankees worse, his fans don't understand how the game of baseball really works, and it was a bad idea to promote him as the ethical savior of the sport." Here's what I think of preregistration: It's a great idea. It's also not the solution to problems of science. I have found preregistration to be useful in my own work. I've seen lots of great work that is not preregistered. I disagree with the claim in the above-linked paper that "Under the guidelines of preregistration, scholars are expected to know what they will find before they run the study; if they get findings they do not expect, they cannot publish them because the study will not be considered legitimate if it was not preregistered." I disagree with that statement in part for the straight-up empirical reason that it's false; there are counterexamples; indeed a couple years ago we discussed a political science study that was preregistered and yielded unexpected findings which were published and were considered legitimate by the journal and the political science profession. More generally, I think of preregistration as a floor, not a ceiling. The preregistered data collection and analysis is what you need to do. In addition, you can do whatever else you want. Preregistration remains overrated if you think it's gonna fix science. Preregistration facilitates the conditions for better science, but if you preregister a bad design, it's still a bad design. Suppose you could go back in time and preregister the collected work of the beauty-and-sex-ratio guy, the ESP guy, and the Cornell Food and Brand Lab guy, and then do all those studies. The result wouldn't be a spate of scientific discoveries; it would just be a bunch of inconclusive results, pretty much no different than the inconclusive results we actually got from that crowd but with the improvement that the inconclusiveness would have been more apparent. As we've discussed before, the benefits of procedural reforms such as preregistration are indirect--making it harder for scientists to fool themselves and others with bad designs--but not direct. Are these indirect benefits greater than the costs? I don't know; maybe McDermott is correct that they're not. I guess it depends on the context. I think preregistration can be valuable, and I say that while recognizing that it's been overrated and inappropriately sold as a miracle cure for scientific corruption. As I wrote a few years ago: In the long term, I believe we as social scientists need to move beyond the paradigm in which a single study can establish a definitive result. In addition to the procedural innovations [of preregistration and mock reports], I think we have to more seriously consider the integration of new studies with the existing literature, going beyond the simple (and wrong) dichotomy in which statistically significant findings are considered as true and nonsignificant results are taken to be zero. But registration of studies seems like a useful step in any case. Derek Jeter was overrated. He was a times a drag on the Yankees' performance. He was still an excellent player and overall was very much a net positive. This entry was posted in Miscellaneous Science, Sociology, Sports, Zombies by Andrew. Bookmark the permalink. 16 thoughts on "What's the difference between Derek Jeter and preregistration?" 1. [986cd85f]Dan Goldstein on September 12, 2022 10:09 AM at 10:09 am said: As said elsewhere "Pre-flight checklists and pre-operative checklists are also unnecessary if you are thinking prudently but they're still good policy" https://twitter.com/dggoldst/status/1567362698282782720 Reply | + [d582]Andrew on September 12, 2022 10:32 AM at 10:32 am said: Yup; well put. Reply | 2. [4bab043b]psyoskeptic on September 12, 2022 11:17 AM at 11:17 am said: Thanks for writing this commentary Andrew. It saves me lots of time answering questions and writing a similar one myself. One group of individuals that need to get on board with what pre-registration is really about is reviewers. You do run into some from time to time who don't get that it's just about openness and not about having to follow the original plan to the letter. But they're perhaps less common than the ones who don't look at pre-registrations at all. Reply | + [d582]Andrew on September 12, 2022 11:49 AM at 11:49 am said: Psyoskeptic: No problem. Youall pay enough for this blog; I should give you value for the money. Reply | 3. [fdbcd85f]chipmunk on September 12, 2022 11:26 AM at 11:26 am said: I believe just about everything outside of social science is "pre-registered", isn't it? If you expect to spend three seasons in the field on an ecological study, you're expected to present the question(s) you expect to answer and show how the data you plan to obtain and the analysis you plan to perform will answer the question(s). If you're lobbying for a mission to Mars, you're expected to explain what you're trying to achieve and specifically how you plan to achieve it. None of that stops researchers from further exploring the existing data, acquiring additional data or performing additional analysis and often it's justified to obtain additional data that doesn't directly bear on the stated questions for a small expense, given that logistics are already paid for. Reply | + [d582]Andrew on September 12, 2022 11:48 AM at 11:48 am said: Chipmunk: No, lots of things outside of social science are not preregistered. I can speak from experience: of the various non-social-science projects here, exactly zero were preregistered. Your confusion might arise in part from terminology. If you want funds for an experiment, whether it be a mission from Mars or a medical experiment, you'll need to write some sort of proposal and give some sort of plan of what you will do, but that plan will not typically be a "pre-registration" in the sense that this term is used. A statement along the lines of, "We plan to study X, Y, and Z, and we have the demonstrated capacity to do the experiment and analyze the data," is not a pregregistration in the sense of being a full protocol for design, data collection, and analysis. Reply | o []jd on September 12, 2022 1:26 PM at 1:26 pm said: >full protocol for design, data collection, and analysis. Well, the proposals for funding that I have worked on involve all of those things...they're basically like a pre-registration in the sense that you seem to use. So how much registration detail constitutes 'pre-registration' in the sense that you use it on this blog? I tried to Google a definition, and found this (among others): "Preregistration, in its simplest form, is a one-page document answering basic questions such as: What question will be studied? What is the hypothesis? What data will be collected, and how will they be analyzed? In its most rigorous form, a "registered report," researchers write an entire paper, minus the results and discussion, and submit it for peer review at a journal, which decides whether to accept it in principle. After the work is completed, reviewers simply check whether the researchers stuck to their own recipe; if so, the paper is published, regardless of what the data show." https://www.science.org/content/article/ more-and-more-scientists-are-preregistering-their-studies-should-you I thought the main purpose was simply to document in advance the inferential goals. https://statmodeling.stat.columbia.edu/2022/08/18/ preregistered-vs-adaptable-bayesian-workflow-and-who-should-do-the-work /#comment-2072391 For example, pre-registering the exact model specification is problematic as pointed out by Betancourt in the quote in the above linked post. However, I think it's useful to state the primary outcomes at least. What amount of detail in preregistration is useful? Reply | # [d582]Andrew on September 12, 2022 2:28 PM at 2:28 pm said: Jd: There are different goals. In no sense should preregistration be thought of as restricting. You don't preregister "the" exact model specification; you preregister "an" exact model specification, and you're always free to fit other models once you've seen the data. 4. [4b06543d]jbayes on September 12, 2022 1:32 PM at 1:32 pm said: Statistics as usually taught, tries to "shed probability". In effect, it takes instances where there's slight opportunity for something and de-facto declares there's no possibility of it happening. A sort of probability laundering. As long as you're equating "slight opportunity" with "no possibility", it creates endless problems. And that's true even if the calculation of "slight opportunity" was done correctly and based on true assumptions, which typically it is not. You can. for example, string a series of such "slight opportunity = no possibility" assertions together and get results that are hopelessly out of wack with reality as the errors build up. Indeed, you can reach any conclusion you want this way. Or you can fish around for an instance of "slight opportunity = no possibility" which leads to an error that suits your purposes. Preregistration, like most solutions to these problems, tries to fix this by artificially limiting the number of cases where a researcher can declare "slight opportunity = no possibility". And in a sense it works. But a better solution would be to go all the way and never make the claim that "slight opportunity =no possibility". In effect, eliminate every opportunity for doing so. Now if only there was a way of doing statistics that propagated all uncertainties throughout, so that if the data suggests something is possible, it always remains a possibility, with no probabilities being "shed" .... Reply | + [d582]Andrew on September 12, 2022 2:26 PM at 2:26 pm said: Jbayes: Yes, what you're referring to at the beginning of your comment is what I refer to as "premature collapsing of the wave function." Reply | 5. [5dcdf28d]Kaiser on September 13, 2022 1:55 AM at 1:55 am said: So in short, some pre-registered designs may be bad designs, and some non-registered designs may be good designs. I want to advocate pre-registering interpretations and actions! I find it a useful defence against people who have pre-conceived conclusions from which they won't budge regardless of evidence. For example, the marketing team believes that spending $$$ on ads will definitely increase sales. Working with statisticians, they pre-register a test to measure the effect of ads on sales. What I push to pre-register in addition to test design is what we would do once the analysis is finished. Of course, if the ads are shown to increase sales, we both agree that they should receive the budget to run more such ads. What if the ads are shown to decrease sales? In pre-registration, we agree that they would stop running these ads. WIthout the preregistration, statisticians are frequently told to keep running analyses until a story can be created to justify continuing the ads despite a negative outcome. Just like pre-registered designs, pre-registering these intentions does not mean that the analyst cannot investigate other interpretations or actions. It's a floor, not a ceiling, as you said. Reply | + [4f3f]name withheld by request on September 13, 2022 8:22 AM at 8:22 am said: Many decades ago, in a previous life, I worked in the field we used to call "computers" but would now be called "IT". I was contacted by a company who wanted to hire me under contract for a few years with a salary far above what I was earning at the time. So I did the interview. They wanted me to prepare an "analysis" of their computing needs in order to justify spending a huge amount of money switching from one mainframe vendor to another. The vendor they wanted to switch to was the company with which I was at the time employed. That's why they wanted to recruit me to do the changeover. Based on what they showed me, there were certainly advantages to making the switch. At least from my point of view as somewhat of an expert in the system they wanted to switch to. But it was an open question whether the advantages were worth the enormous cost of revamping their entire computing infrastructure company-wide. So I had to ask "elephant in the room" question. What if my analysis showed that the benefits of switching were not sufficient to justify the change? Wouldn't I be out of a job after just a year or so? The person I was interviewing with patiently explained that by hiring me to do the analysis they were confident my own self-interest would guarantee I'd come up with the right answer. So OK, bonus points for honesty but it just all smelled fishy to me. Today I routinely encounter research that has been peer reviewed and published where the same sort of incentives strongly apply to both the researchers and the peer reviewers. Guess I missed the boat all those years ago and should have just taken the money and done a very well paid 3-4 year gig that I could leverage into another similar position at the next company. Reply | 6. [32b4b55a]Dace on September 13, 2022 3:19 PM at 3:19 pm said: Have you seen "The Laws of Science" by Niko McCarty (https:// newscience.substack.com/p/laws-of-science)? He discusses preregistration in some detail, but in particular, his very first figure looks (results of trials with and without mandatory registration) pretty alarming. (It was reproduced from Kaplan R.M. & Irvin V.L. PLOS ONE (2015)) Reply | + [d582]Andrew on September 13, 2022 4:51 PM at 4:51 pm said: Dace: I like some of that post by McCarty but I think he's a bit too idealistic about the legal system, which he is using as his comparison point. Reply | 7. [d1deed36]Mark on September 13, 2022 4:43 PM at 4:43 pm said: McDermott is building an army of strawmen. Reply | + [d582]Andrew on September 13, 2022 4:48 PM at 4:48 pm said: Mark: My (uninformed) guess is that McDermott is annoyed at the grand claims that have been made for preregistration and is reacting by disparaging the idea, in the same way that people were annoyed at the Derek Jeter hype and followed up by underrating him. Reply | Leave a Reply Cancel reply Your email address will not be published. Required fields are marked * [ ] [ ] [ ] [ ] [ ] [ ] [ ] Comment * [ ] Name [ ] Email [ ] Website [ ] [Post Comment] [ ] [ ] [ ] [ ] [ ] [ ] [ ] D[ ] * Art * Bayesian Statistics * Causal Inference * Decision Theory * Economics * Jobs * Literature * Miscellaneous Science * Miscellaneous Statistics * Multilevel Modeling * Papers * Political Science * Public Health * Sociology * Sports * Stan * Statistical computing * Statistical graphics * Teaching * Zombies 1. Joshua on "Merchants of doubt" operating within organized science September 14, 2022 3:11 PM Anoneuoid - > "healthy vaccinee effect" was another one... It's noteworthy how often you appeal to your own authority to... 2. Raghu Parthasarathy on Why do Dickens novels have all those coincidences?September 14, 2022 2:26 PM "And finally, we are told of 'the sin, most gross, most palpable, which Dickens everywhere commits in his abuse of... 3. gwern on Why do Dickens novels have all those coincidences? September 14, 2022 1:49 PM My theory, based on picking up _The Pickwick Papers_ and getting 50 pieces of paper in before putting it down... 4. Andrew on Why do Dickens novels have all those coincidences? September 14, 2022 12:54 PM David: Thanks for the recommendation. I read London and the South-East, and I absolutely loved it. 5. David on Why do Dickens novels have all those coincidences? September 14, 2022 12:38 PM Have you read the novella Turbulence by David Szalay? It fits exactly your theory about coincidences as a way to... 6. Kevin Nelson on Why do Dickens novels have all those coincidences?September 14, 2022 11:27 AM If you think Charles Dickens has a lot of coincidences, you should try reading Edgar Rice Burroughs. Anyway, I'm pretty... 7. chipmunk on Why do Dickens novels have all those coincidences? September 14, 2022 10:59 AM The reference to Harry Potter in Rees' piece begs a question: Is it appropriate to compare modern literary fiction with... 8. Matt on Geoff Dyer Kazuo Ishiguro Owen Sheers David Leavitt Veronica GengSeptember 14, 2022 10:09 AM Forgot to add Alan Moorehead: https://archive.clivejames.com/ books/moore.htm 9. Andrew on Why do Dickens novels have all those coincidences? September 14, 2022 10:08 AM Tomas: Your explanation doesn't work, because the underlying question is not, Why are published books full of coincidences, but, rather,... 10. Adede on Why do Dickens novels have all those coincidences? September 14, 2022 10:07 AM I think it's a pretty big coincidence that Superman is always within a couple of blocks of every supervillain attack. 11. Tamas K Papp on Why do Dickens novels have all those coincidences?September 14, 2022 9:57 AM Publication bias. Books/plots with some coincidences are more interesting, especially when it is about characters we have invested in. It... 12. Anoneuoid on "Merchants of doubt" operating within organized scienceSeptember 14, 2022 9:04 AM OK I see what they've done. Unfortunately, you clearly still do not. It is fascinating how people can write so... 13. Andrew on Geoff Dyer Kazuo Ishiguro Owen Sheers David Leavitt Veronica GengSeptember 14, 2022 6:21 AM Matt: I dunno, something about Edmund Wilson . . . I'm interested in the topics he writes about, but I... 14. Matt Broughton on Geoff Dyer Kazuo Ishiguro Owen Sheers David Leavitt Veronica GengSeptember 14, 2022 6:00 AM For fiction, I would add Kingsley Amis, Christopher Isherwood and Norman Mailer in "The Executioner's Song." For non-fiction, Edmund Wilson,... 15. Matt Skaggs on "Merchants of doubt" operating within organized scienceSeptember 14, 2022 1:16 AM Chris wrote: "But there unquestionably are "positive feedbacks" in the climate system, as indicated in the water vapour/CO2 effect above.... 16. Phil on "You should always (always) have a Naive model. It's the simplest, cleanest, most intuitive way to explain whether your system is at least treading water. And if it is (that's a big IF), how much better than Naive is it."September 13, 2022 6:55 PM Yes 17. Anonymous on A garland of retractions for the Ohio State Department of [S:Chutzpah:S] Cancer Biology and GeneticsSeptember 13, 2022 6:40 PM Search the first author's name on http://www.fcmcclerk.com/case/ search . 18. Publius on Seth Roberts and political polarizationSeptember 13, 2022 5:38 PM "What's the difference between Bush's disinformation about Iraq and QAnon?" As someone who saw the QAnon movement from the beginning,... 19. Andrew on What's the difference between Derek Jeter and preregistration?September 13, 2022 4:51 PM Dace: I like some of that post by McCarty but I think he's a bit too idealistic about the legal... 20. Andrew on What's the difference between Derek Jeter and preregistration?September 13, 2022 4:48 PM Mark: My (uninformed) guess is that McDermott is annoyed at the grand claims that have been made for preregistration and... 21. Mark on What's the difference between Derek Jeter and preregistration?September 13, 2022 4:43 PM McDermott is building an army of strawmen. 22. Andrew on Geoff Dyer Kazuo Ishiguro Owen Sheers David Leavitt Veronica GengSeptember 13, 2022 4:12 PM Ja hoor. 23. Jonathan (another one) on Geoff Dyer Kazuo Ishiguro Owen Sheers David Leavitt Veronica GengSeptember 13, 2022 4:09 PM If I knew Norwegian, I'd say "Stikk den opp i rumpa." Apparently, that's fairly direct prose. 24. Dace on What's the difference between Derek Jeter and preregistration?September 13, 2022 3:19 PM Have you seen "The Laws of Science" by Niko McCarty (https:// newscience.substack.com/p/laws-of-science)? He discusses preregistration in some detail, but in particular,... 25. Joshua on "Merchants of doubt" operating within organized science September 13, 2022 3:18 PM Anoneuoid - Are you comparing direct replications to conceptual replications to argue for the former in comparison to the latter? ... 26. Chris on "Merchants of doubt" operating within organized science September 13, 2022 3:15 PM OK I see what they've done. They've used the Stefan-Boltzmann eq. to calculate the blackbody temperature of an atmosphereless planet... 27. Eric Novik on Geoff Dyer Kazuo Ishiguro Owen Sheers David Leavitt Veronica GengSeptember 13, 2022 3:02 PM I love Murakami's vivid writing style but there is so much mysticism that I can't quite suspend disbelief. Hayao Miyazaki... 28. Anoneuoid on "Merchants of doubt" operating within organized scienceSeptember 13, 2022 3:02 PM Ok, well here is the link again: https:// physics.stackexchange.com/questions/508573/ why-does-this-simple-equation-predict-the-venus-surface-temperature-so-accuratel It should be obvious to anyone reading it that it is about... 29. Chris on "Merchants of doubt" operating within organized science September 13, 2022 2:48 PM Nope, I'm pretty sure that they're calculating the temperature at the top of the atmosphere. 30. Andrew on Geoff Dyer Kazuo Ishiguro Owen Sheers David Leavitt Veronica GengSeptember 13, 2022 2:38 PM Jonathan: All the rest of our blog commenters read Norwegian so I don't know what your problem is. 31. Savoy on "Merchants of doubt" operating within organized science September 13, 2022 2:32 PM Joshua: Thanks for replying! I would imagine that extreme sea level rise should be pretty easy to test empirically (in... 32. Anoneuoid on "Merchants of doubt" operating within organized scienceSeptember 13, 2022 1:56 PM The problem is we *can* calculate the surface temperatures without including albedo, etc. With *zero* free parameters, and derived from... 33. Chris on "Merchants of doubt" operating within organized science September 13, 2022 1:39 PM Anoneuoid - ok I we may have to agree not so much to differ but to decide that it's too... 34. Chris on "Merchants of doubt" operating within organized science September 13, 2022 1:28 PM Matt - sure, no problem. "Positive feedback" can have different meanings. For example the meaning of the term in electronic... 35. Anoneuoid on "Merchants of doubt" operating within organized scienceSeptember 13, 2022 1:24 PM Anoneuoid - you linked to a page describing the use of the Stefan Boltzmann (SB) equation. Isn't that sufficient to... 36. Jonathan (another one) on Geoff Dyer Kazuo Ishiguro Owen Sheers David Leavitt Veronica GengSeptember 13, 2022 1:09 PM I agree. I'd also nominate Knausgaard, though I don't read Norwegian, and it's not clear how much is the translation... 37. Chris on "Merchants of doubt" operating within organized science September 13, 2022 1:07 PM Anoneuoid - you linked to a page describing the use of the Stefan Boltzmann (SB) equation. Isn't that sufficient to... 38. Matt Skaggs on "Merchants of doubt" operating within organized scienceSeptember 13, 2022 12:55 PM Chris wrote: ""positive feedback" in climate research is not equivalent to "positive feedback" in, for example, engineering" OK. Wow. I... 39. Kyle C on Geoff Dyer Kazuo Ishiguro Owen Sheers David Leavitt Veronica GengSeptember 13, 2022 12:33 PM Roberto Bolan[y]o's translator, Natasha Wimmer, renders his prose like this too, although I don't read Spanish and can't judge the... 40. Andrew on "Merchants of doubt" operating within organized science September 13, 2022 11:58 AM Savoy: "Bourgeoisie experts"?? C'mon. Save that sort of plain-man posturing for twitter, 4chan, or the meeting of the local Socialist... 41. OliP on Geoff Dyer Kazuo Ishiguro Owen Sheers David Leavitt Veronica GengSeptember 13, 2022 11:53 AM Didn't early Orwell aim for some purple prose? I seem to recall that he had some such aim to combine... 42. Dan Dillon on Geoff Dyer Kazuo Ishiguro Owen Sheers David Leavitt Veronica GengSeptember 13, 2022 10:07 AM I would add Haruki Murakami--his stories often involve a delightful does of magic realism, but it feels like you're engaged... 43. David in Tokyo on "Merchants of doubt" operating within organized scienceSeptember 13, 2022 10:07 AM "To iterate is human, to recurse divine." Guy L. Steele. 44. Daniel Hadas on Geoff Dyer Kazuo Ishiguro Owen Sheers David Leavitt Veronica GengSeptember 13, 2022 9:52 AM That's wonderful on Pauline Kael. 45. Joshua on "Merchants of doubt" operating within organized science September 13, 2022 9:14 AM There's a guy who posts a lot about his earth-shattering theory that thunderstorms prove the "homeostasis" of the planet, in... 46. Joshua on "Merchants of doubt" operating within organized science September 13, 2022 8:46 AM Comparing the homeostasis of the temperature of the human body to the stability of the temperature of the planet has... 47. Anoneuoid on "Merchants of doubt" operating within organized scienceSeptember 13, 2022 8:45 AM Is there evidence that the Earth has such control systems? I don't think so. The evidence is the existence of... 48. Joshua on "Merchants of doubt" operating within organized science September 13, 2022 8:26 AM Savoy - The projections you speak of (the aren't predictions, BTW) are based on averages. Do it seems to me... 49. name withheld by request on What's the difference between Derek Jeter and preregistration?September 13, 2022 8:22 AM Many decades ago, in a previous life, I worked in the field we used to call "computers" but would now... 50. Savoy on "Merchants of doubt" operating within organized science September 13, 2022 5:32 AM Can any of the bourgeoisie experts here tell me: Won't we soon find out the truth about the strength of... Proudly powered by WordPress