https://statmodeling.stat.columbia.edu/2023/12/23/bayesians-moving-from-defense-to-offense-i-really-think-its-kind-of-irresponsible-now-not-to-use-the-information-from-all-those-thousands-of-medical-trials-that-came-before-is-that-very/ Skip to primary content Statistical Modeling, Causal Inference, and Social Science Search [ ] [Search] Main menu * Home * Authors * Blogs We Read * Sponsors Post navigation Plagiarism means never having to say you're clueless. Bayesians moving from defense to offense: "I really think it's kind of irresponsible now not to use the information from all those thousands of medical trials that came before. Is that very radical?" Posted on December 23, 2023 9:45 AM by Andrew [Screenshot-2023-12-22-at-16] Erik van Zwet, Sander Greenland, Guido Imbens, Simon Schwab, Steve Goodman, and I write: We have examined the primary efficacy results of 23,551 randomized clinical trials from the Cochrane Database of Systematic Reviews. We estimate that the great majority of trials have much lower statistical power for actual effects than the 80 or 90% for the stated effect sizes. Consequently, "statistically significant" estimates tend to seriously overestimate actual treatment effects, "nonsignificant" results often correspond to important effects, and efforts to replicate often fail to achieve "significance" and may even appear to contradict initial results. To address these issues, we reinterpret the P value in terms of a reference population of studies that are, or could have been, in the Cochrane Database. This leads to an empirical guide for the interpretation of an observed P value from a "typical" clinical trial in terms of the degree of overestimation of the reported effect, the probability of the effect's sign being wrong, and the predictive power of the trial. Such an interpretation provides additional insight about the effect under study and can guard medical researchers against naive interpretations of the P value and overoptimistic effect sizes. Because many research fields suffer from low power, our results are also relevant outside the medical domain. Also this new paper from Zwet with Lu Tian and Rob Tibshirani: Evaluating a shrinkage estimator for the treatment effect in clinical trials The main objective of most clinical trials is to estimate the effect of some treatment compared to a control condition. We define the signal-to-noise ratio (SNR) as the ratio of the true treatment effect to the SE of its estimate. In a previous publication in this journal, we estimated the distribution of the SNR among the clinical trials in the Cochrane Database of Systematic Reviews (CDSR). We found that the SNR is often low, which implies that the power against the true effect is also low in many trials. Here we use the fact that the CDSR is a collection of meta-analyses to quantitatively assess the consequences. Among trials that have reached statistical significance we find considerable overoptimism of the usual unbiased estimator and under-coverage of the associated confidence interval. Previously, we have proposed a novel shrinkage estimator to address this "winner's curse." We compare the performance of our shrinkage estimator to the usual unbiased estimator in terms of the root mean squared error, the coverage and the bias of the magnitude. We find superior performance of the shrinkage estimator both conditionally and unconditionally on statistical significance. Let me just repeat that last sentence: We find superior performance of the shrinkage estimator both conditionally and unconditionally on statistical significance. From a Bayesian standpoint, this is no surprise. Bayes is optimal if you average over the prior distribution and can be reasonable if averaging over something close to the prior. Especially reasonable in comparison to naive unregularized estimates (as here). Erik summarizes: We've determined how much we gain (on average over the Cochrane Database) by using our shrinkage estimator. It turns out to be about a factor 2 more efficient (in terms of the MSE) than the unbiased estimator. That's roughly like doubling the sample size! We're using similar methods as our forthcoming paper about meta-analysis with a single trial. People sometimes ask me how I've changed as a statistician over the years. One answer I've given is that I've gradually become more Bayesian. I started out as a skeptic, concerned about Bayesian methods at all; then in grad school I started using Bayesian statistics in applications and realizing it could solve some problems for me; when writing BDA and ARM, still having the Bayesian cringe and using flat priors as much as possible, or not talking about priors at all; then with Aleks, Sophia, and others moving toward weakly informative priors; eventually under the influence of Erik and others trying to use direct prior information. At this point I've pretty much gone full Lindley. Just as a comparison to where my colleagues and I are now, check out my response in 2008 to a question from Sanjay Kaul about how to specify a prior distribution for a clinical trial. I wrote: I suppose the best prior distribution would be based on a multilevel model (whether implicit or explicit) based on other, similar experiments. A noninformative prior could be ok but I prefer something weakly informative to avoid your inferences being unduly affected by extremely unrealistic possibilities in the tail of the distribuiton. Nothing wrong with this advice, exactly, but I was still leaning in the direction of noninformativeness in a way that I would not anymore. Sander Greenland replied at the time with a recommendation to use direct prior information. (And, just for fun, here's a discussion from 2014 on a topic where Sander and I disagree.) Erik concludes: I really think it's kind of irresponsible now not to use the information from all those thousands of medical trials that came before. Is that very radical? That last question reminds me of our paper from 2008, Bayes: Radical, Liberal, or Conservative? P.S. Also this: [Screenshot-2023-12-22-at-16] You can click through to see the whole story. This entry was posted in Bayesian Statistics, Multilevel Modeling, Public Health by Andrew. Bookmark the permalink. 6 thoughts on "Bayesians moving from defense to offense: "I really think it's kind of irresponsible now not to use the information from all those thousands of medical trials that came before. Is that very radical?"" 1. [e6d70454]Deborah Mayo on December 23, 2023 12:11 PM at 12:11 pm said: Andrew: If you've gone "full Lindley", you've become a subjective Bayesian, adopting the very philosophy you used to strenuously reject. Error statisticians use background knowledge but without become subjectivists, or denying the importance of error probabilities. A link to Lindley's statistical philosophy on my blog is here: https://errorstatistics.com/2012/07/12/ dennis-lindleys-philosophy-of-statistics/ Reply | + [d582]Andrew on December 23, 2023 3:34 PM at 3:34 pm said: Deborah: Yeah, I was exaggerating with the "full Lindley" bit. Reply | + [0358]Henry Wyneken on December 23, 2023 5:10 PM at 5:10 pm said: I think this is a useful tool, but I'm not yet sure when it should be used. I've used shrinkage estimators in industry. To keep it hypothetical, let's say my job was to predict sales for a portfolio of products. In that situation, we do care about minimizing the overall prediction loss. I'm imagining someone who works at the FDA looking at the landscape of all drug trials and saying "well, we know how the distribution of SNRs looks like, so I will use that to rationally adjust my assessment of this new trial". I'm also thinking of this quote from van Zwet, Tian and Tibshirani: "The hazard ratio was estimated at 0.75 with 95% confidence interval of (0.55, 1.02). However, it has been established that many trials have a low signal-to-noise ratio, which can lead to upward bias in the estimate of the hazard ratio. If we apply this general information to our particular trial, the hazard ratio estimate becomes 0.84 with interval (0.62, 1.07)." I don't think this kind of statement is very compatible with a falsificationist approach to science. If my theory predicts effect size eta*, and my competitors have estimated effect sizes {eta1, eta2, ..., eta5}, I want to design an experiment that has a high chance of being close to eta* and far away from {eta1,...,eta5}, if that's the truth. But if I take a shrinkage approach, I am in effect hamstrung by the prevailing theory! I am forced to shrink my estimates closer to theirs. One could object: "why should your one study matter much in comparison to the five?" My answer to that is my study is qualitatively better than theirs - we make weaker assumptions, or we have better measurements. I think that in this context, shrinkage penalizes outsider viewpoints too harshly, in the same way that allowing researcher degrees of freedom can give outsiders too much freedom. From a metascience point of view, I think it would be better to allow outsiders to use unbiased estimators, provided that they come by them honestly. Reply | 2. [268519d0]Jonathan (another one) on December 23, 2023 2:11 PM at 2:11 pm said: Is there a non-paywalled version? Reply | + [91ab]Erik on December 23, 2023 2:21 PM at 2:21 pm said: https://evidence.nejm.org/stoken/default+domain/ KXCEVZP8G9ITKFMSSDMZ/full?redirectUri=/doi/full/10.1056/ EVIDoa2300003 Reply | + [d582]Andrew on December 23, 2023 3:35 PM at 3:35 pm said: Here's the non-paywalled preprint! Reply | Leave a Reply Cancel reply Your email address will not be published. Required fields are marked * [ ] [ ] [ ] [ ] [ ] [ ] [ ] Comment * [ ] Name [ ] Email [ ] Website [ ] [Post Comment] [ ] [ ] [ ] [ ] [ ] [ ] [ ] D[ ] * Art * Bayesian Statistics * Causal Inference * Decision Analysis * Economics * Jobs * Literature * Miscellaneous Science * Miscellaneous Statistics * Multilevel Modeling * Papers * Political Science * Public Health * Sociology * Sports * Stan * Statistical Computing * Statistical Graphics * Teaching * Zombies 1. Chris on Plagiarism means never having to say you're clueless. December 23, 2023 5:18 PM That doesn't seem correct Matt. George Mason Provost Peter Stearns in his letter to the Faculty (22 Feb, 2012) writes:... 2. Henry Wyneken on Bayesians moving from defense to offense: "I really think it's kind of irresponsible now not to use the information from all those thousands of medical trials that came before. Is that very radical?"December 23, 2023 5:10 PM I think this is a useful tool, but I'm not yet sure when it should be used. I've used shrinkage... 3. Anonymous on Plagiarism means never having to say you're clueless.December 23, 2023 4:40 PM I am familiar with doing that, and acknowleding peer-reviewers in this way. To me, it's not specific enough regarding me... 4. Phil on Plagiarism means never having to say you're clueless. December 23, 2023 3:45 PM chipmunk, I think you've missed my main point, which is not that Claudine Gay is not guilty of something serious... 5. Andrew on Bayesians moving from defense to offense: "I really think it's kind of irresponsible now not to use the information from all those thousands of medical trials that came before. Is that very radical?"December 23, 2023 3:35 PM Here's the non-paywalled preprint! 6. Andrew on Bayesians moving from defense to offense: "I really think it's kind of irresponsible now not to use the information from all those thousands of medical trials that came before. Is that very radical?"December 23, 2023 3:34 PM Deborah: Yeah, I was exaggerating with the "full Lindley" bit. 7. Andrew on Plagiarism means never having to say you're clueless. December 23, 2023 3:31 PM Matt: Nobody's being clever here, and certainly not Wegman, who copied without attribution in multiple papers, not just that one.... 8. Andrew on Plagiarism means never having to say you're clueless. December 23, 2023 3:28 PM Anon: When I publish a paper that has made use of reviewers' ideas or wording, I add something like "and... 9. Matt Skaggs on Plagiarism means never having to say you're clueless.December 23, 2023 3:21 PM "Wegman was sanctioned by his University for unprofessional practices, had a letter of reprimand put on his file and had... 10. Carlos on Plagiarism means never having to say you're clueless. December 23, 2023 2:56 PM If you're interested in plagiarism in other media which goes deep into the culture of plagiarism in Youtube, take a... 11. Erik on Bayesians moving from defense to offense: "I really think it's kind of irresponsible now not to use the information from all those thousands of medical trials that came before. Is that very radical?"December 23, 2023 2:21 PM https://evidence.nejm.org/stoken/default+domain/ KXCEVZP8G9ITKFMSSDMZ/full?redirectUri=/doi/full/10.1056/ EVIDoa2300003 12. Jonathan (another one) on Bayesians moving from defense to offense: "I really think it's kind of irresponsible now not to use the information from all those thousands of medical trials that came before. Is that very radical?"December 23, 2023 2:11 PM Is there a non-paywalled version? 13. Joshua on Plagiarism means never having to say you're clueless. December 23, 2023 1:44 PM W/r/t different definitions of plagiarism: > In both cases, modern plagiarism software was used to detect repetition of rhetoric --... 14. Allen Schmaltz on Explainable AI works, but only when we don't need itDecember 23, 2023 12:47 PM In practice for high-dimensional inputs, I would argue that three things are needed, and just presenting a probability alone has... 15. Joshua on Plagiarism means never having to say you're clueless. December 23, 2023 12:46 PM Chris - Agreed. Good distinction. Re that Wegman example, yikes. I'd say it's impossible to find a good faith, innocent... 16. Chris on Plagiarism means never having to say you're clueless. December 23, 2023 12:43 PM The point is that many allegations of plagiarism have little to do with attribution and everything to do with exploiting... 17. Anonymous on Plagiarism means never having to say you're clueless.December 23, 2023 12:35 PM I looked up what "plagiarism" exactly means or implies and came across the following description: "Plagiarism is the use of... 18. Deborah Mayo on Bayesians moving from defense to offense: "I really think it's kind of irresponsible now not to use the information from all those thousands of medical trials that came before. Is that very radical?"December 23, 2023 12:11 PM Andrew: If you've gone "full Lindley", you've become a subjective Bayesian, adopting the very philosophy you used to strenuously reject.... 19. Chris on Plagiarism means never having to say you're clueless. December 23, 2023 11:50 AM Do need to consider the context though and there is a fundamental difference between student plagiarism and academic plagiarism. Students... 20. Anonymous on Plagiarism means never having to say you're clueless.December 23, 2023 10:58 AM If it might be the case that certain people in science get way more attention when they say or write... 21. Joshua on Plagiarism means never having to say you're clueless. December 23, 2023 10:52 AM Andrew - I'm definitely in general agreement! One of the hats I've worn is helping students with their writing (proving... 22. Nico on Explainable AI works, but only when we don't need it December 23, 2023 10:28 AM "... and the same explanation technique is often highly sensitive to small differences in the function to be explained (e.g.,... 23. Andrew on Plagiarism means never having to say you're clueless. December 23, 2023 10:13 AM Joshua: As I wrote in my above post, in theory I agree that it should be possible to plagiarize without... 24. Joshua on Plagiarism means never having to say you're clueless. December 23, 2023 10:10 AM Chipmunk - Did did someone wearing a CNN or NYT T-shirt steal your candy when you were a kid? 25. Joshua on Plagiarism means never having to say you're clueless. December 23, 2023 9:58 AM But that is an example where plagiarism, in a technical sense doesn't necessarily indicate a lack of understanding. I think... 26. Joshua on Plagiarism means never having to say you're clueless. December 23, 2023 9:52 AM Phil - . My reaction was the similar. I'm generally in strong agreement with Andrew's point. However I think it's... 27. Andrew on Plagiarism means never having to say you're clueless. December 23, 2023 9:30 AM Rahul: I think a key difference here is that in an academic book or article, it's easy to use quotation... 28. Anoneuoid on Plagiarism means never having to say you're clueless.December 23, 2023 8:29 AM Is Rahul's presentation deficient because he hasn't dived into whatever depths of whatever putative controversies there might (or might not)... 29. Dale Lehman on Plagiarism means never having to say you're clueless.December 23, 2023 7:37 AM I've often used other people's material in classes. I'm pretty sloppy about attribution - sometimes I make sure to cite... 30. Chris on Plagiarism means never having to say you're clueless. December 23, 2023 7:05 AM IMO attribution is the issue. Rahul's example is pertinent. Rahul (apols for referring to you in the third person) very... 31. Anoneuoid on Plagiarism means never having to say you're clueless.December 23, 2023 6:05 AM Attribution isn't the real issue. The problem is that you are taking whatever explanation out of context. That is what... 32. Chris on Plagiarism means never having to say you're clueless. December 23, 2023 4:36 AM That's not plagiarism. There was a time when you could be considered to be flouting copyright law, but rules have... 33. Rahul on Plagiarism means never having to say you're clueless. December 23, 2023 3:58 AM Andrew: Here's one example: while teaching an applied engineering class I often assemble a slide deck or handouts and often... 34. chipmunk on Plagiarism means never having to say you're clueless. December 23, 2023 3:22 AM Phil: no doubt the NYT, known for its dogged pursuit of lefty skullduggery, is presenting this case in the ugliest... 35. Karl William Smith on Plagiarism means never having to say you're clueless.December 22, 2023 10:01 PM The only thing I would add is that in the case where you don't understand attribution can make you look... 36. Elin on This empirical paper has been cited 1616 times but I don't find it convincing. There's no single fatal flaw, but the evidence does not seem so clear. How to think about this sort of thing? What to do? First, accept that evidence might not all go in one direction. Second, make lots of graphs. Also, an amusing story about how this paper is getting cited nowadays.December 22, 2023 6:59 PM I skimmed the original paper, which is, as we know, by 3 economists. They ask: "First, does public financing affect... 37. Phil on Plagiarism means never having to say you're clueless. December 22, 2023 6:44 PM There's plagiarism and then there's plagiarism. What I mean is: it seems that sometimes people plagiarize because they aren't confident... 38. Wonks Anonymous on Plagiarism means never having to say you're clueless.December 22, 2023 6:25 PM I've read a number of posts about plagiarism here, but I don't recall Andrew previously discussing being plagiarized himself. I'd... 39. Wonks Anonymous on Plagiarism means never having to say you're clueless.December 22, 2023 6:21 PM Claudine Gay plagiarizing an acknowledgements section is funnier than any academic plagiarism case I'd heard of before. Using the logic... 40. Rahul on Plagiarism means never having to say you're clueless. December 22, 2023 5:42 PM Wow. It's been ten years?! Time flies. 41. Anoneuoid on Plagiarism means never having to say you're clueless.December 22, 2023 5:30 PM I never got an answer as to whether we should cite Copernicus if we write that the earth revolves around... 42. Daniel Weissman on Plagiarism means never having to say you're clueless.December 22, 2023 5:16 PM I actually have run into something along the lines of Rahul's example a number of times. There are some great... 43. Andrew on Plagiarism means never having to say you're clueless. December 22, 2023 5:06 PM To put it another way: I have no doubt that you're sincere. Wegman I'm not so sure about. I don't... 44. Andrew on Plagiarism means never having to say you're clueless. December 22, 2023 4:41 PM Matt: 1. You can cite Copernicus if you'd like! If you directly quote from Copernicus, I think you should definitely... 45. Matt Skaggs on Plagiarism means never having to say you're clueless.December 22, 2023 4:31 PM I never got an answer as to whether we should cite Copernicus if we write that the earth revolves around... 46. Dale Lehman on Plagiarism means never having to say you're clueless.December 22, 2023 3:35 PM Somewhat tangential: attorneys often discourage the use of direct quotes in expert witness testimony. There is the danger that whatever/whoever... 47. Andrew Wilson on Plagiarism means never having to say you're clueless.December 22, 2023 3:28 PM Interesting perspective. Plagiarism really does seem to indicate lack of passion and curiosity for the subject matter. I can see... 48. Anoneuoid on Plagiarism means never having to say you're clueless.December 22, 2023 2:41 PM I think you're right Andrew, it is a symptom of lacking understanding. Plagiarism erases the original context of whatever ideas... 49. Giacomo Petrillo on Explainable AI works, but only when we don't need itDecember 22, 2023 2:05 PM My impression is that the frontier of this kind of research has been for a while the work of Chris... 50. Andrew on Plagiarism means never having to say you're clueless. December 22, 2023 2:00 PM Ben: A bunch of people were emailing me with questions about plagiarism. Regarding the Claudine Gay case, I don't really... Proudly powered by WordPress