https://statmodeling.stat.columbia.edu/2023/02/25/not-within-spitting-distance-challenges-in-measuring-ovulation-as-an-example-of-the-general-issue-of-the-importance-of-measurement-in-statistics/ Skip to primary content Statistical Modeling, Causal Inference, and Social Science Search [ ] [Search] Main menu * Home * Authors * Blogs We Read * Sponsors Post navigation with Lauren Kennedy and Jessica Hullman: "Causal quartets: Different ways to attain the same average treatment effect" "Not within spitting distance": Challenges in measuring ovulation, as an example of the general issue of the importance of measurement in statistics Posted on February 25, 2023 9:30 AM by Andrew Ruben Arslan writes: I don't know how interested you still are in what's going on in ovulation research, but I hoped you might find the attached piece interesting. Basically, after the brouhaha following Harris et al. 2013 observation that the same research groups used very heterogeneous definitions of the fertile window, the field moved towards salivary steroid immunoassays as a hormonal index of cycle phase. Turns out that this may not have improved matters, as these assays do not actually index cycle phase very well at all, as we show. In fact, these "measures" are probably outperformed by imputation from cycle phase measures based on LH surges or counting from the next menstrual onset. I think the preregistration revolution helped, because without researcher flexibility to save the day a poor measure is a bigger liability. But it still took too long to realize given how poor the measures seem to be. You wouldn't be able to "predict" menstruation with these assays with much accuracy, let alone ovulation. The models were estimated in Stan via brms. I'd be interested to hear what you or your commenters have to say about some of the more involved steps I took to guesstimate the unmeasured correlation between salivary and serum steroids. I think the field is changing for the better -- almost everyone I approached shared their data for this project (much of it was public already) and though the conclusions are hard to accept, most did. The preprint is here. This is a good example of the challenges and importance of measurement in a statistical study. Earlier we've discussed the general issue and various special cases such as the claim that North Korea is more democratic than North Carolina. My hypothesis on all this is that when students are taught research methods, they're taught about statistical analysis and a bit about random sampling. Then when they do research, they're super-aware of statistical issues such as how to calculate a standard error and also aware of issues regarding sampling, experimentation, and random assignment--but they don't usually think of measurement as a statistical/methods/research challenge. They just take some measurement and run with it, without reflecting on whether it makes sense, let alone studying its properties of reliability and validity. Then if they get statistical significance, they think they've made a discovery, and that's a win, so game over. I don't really know where to start to help fix this problem. But gathering some examples is a start. Maybe we need to write a paper with a title such as Measurement Matters, including a bunch of these examples. Publish it as an editorial in Science or Nature and maybe it will have some effect? Arslan adds: The paper is now published after three rounds of reviews. One interesting bit that peer review added: a reviewer didn't quite trust my LOO-R estimates of the imputation accuracy and wanted me to say they were unvalidated and would probably be lower in independent data. So, I added a sanity check with independent data. The correlations were within +-0.01 of the LOO-R estimates. Pretty impressive job, LOO and team. Leave-one-out cross validation FTW! This entry was posted in Bayesian Statistics, Miscellaneous Statistics, Teaching, Zombies by Andrew. Bookmark the permalink. 1 thought on ""Not within spitting distance": Challenges in measuring ovulation, as an example of the general issue of the importance of measurement in statistics" 1. [fdbcd85f]chipmunk on February 25, 2023 10:28 AM at 10:28 am said: "They just take some measurement and run with it, without reflecting on whether it makes sense, let alone studying its properties of reliability and validity." Yes, I strongly agree! Most of the work in the Unbelievable Results category fails before the data analysis even starts because the measurement method has untenable assumptions or simply doesn't measure what it purports to measure. It's a problem with people doing "data analysis" instead of science. For science you need to establish the method and test it multiple times before you deploy it. Reply | Leave a Reply Cancel reply Your email address will not be published. Required fields are marked * [ ] [ ] [ ] [ ] [ ] [ ] [ ] Comment * [ ] Name [ ] Email [ ] Website [ ] [Post Comment] [ ] [ ] [ ] [ ] [ ] [ ] [ ] D[ ] * Art * Bayesian Statistics * Causal Inference * Decision Theory * Economics * Jobs * Literature * Miscellaneous Science * Miscellaneous Statistics * Multilevel Modeling * Papers * Political Science * Public Health * Sociology * Sports * Stan * Statistical computing * Statistical graphics * Teaching * Zombies 1. turn_of_the_90s on with Lauren Kennedy and Jessica Hullman: "Causal quartets: Different ways to attain the same average treatment effect"February 25, 2023 5:24 PM I'm curious why you plotted a solid line at y=0 or, at least, did not include an additional line -... 2. Joe Nadeau on with Lauren Kennedy and Jessica Hullman: "Causal quartets: Different ways to attain the same average treatment effect"February 25, 2023 5:05 PM When fighting et al like that is discovered, it is a strict requirement to separate the mice. It is considered... 3. Alan Goldhammer on David Bowie vs. Beverly Cleary; Dahl advances February 25, 2023 3:24 PM This one is such a toss up as both have lots of good reasons to advance. I've voted for each... 4. Gib Bassett on with Lauren Kennedy and Jessica Hullman: "Causal quartets: Different ways to attain the same average treatment effect"February 25, 2023 1:01 PM Inspired by your Anscombe (1973) reference: Conditional quantiles along with the conditional expectation. https:// gib.people.uic.edu/Anscombe.pdf 5. Ben on How to think about proofs of correctness of computer programs?February 25, 2023 12:36 PM That DLP paper was interesting. It brings to mind: * How handy the Python type system is even though the... 6. Dzhaughn on How to think about proofs of correctness of computer programs?February 25, 2023 12:33 PM As usual, the discussion of this topic starts from the wrong premise. (Although Henning above is onto the problem, so... 7. chipmunk on "Not within spitting distance": Challenges in measuring ovulation, as an example of the general issue of the importance of measurement in statisticsFebruary 25, 2023 10:28 AM "They just take some measurement and run with it, without reflecting on whether it makes sense, let alone studying its... 8. Anoneuoid on with Lauren Kennedy and Jessica Hullman: "Causal quartets: Different ways to attain the same average treatment effect"February 25, 2023 8:38 AM Sorry, I should have been more explicit. They need to check whether the mice are fighting (even killing) each other,... 9. Joe Nadeau on with Lauren Kennedy and Jessica Hullman: "Causal quartets: Different ways to attain the same average treatment effect"February 25, 2023 8:00 AM Agreed, the study has many deficiencies, including poor documentation of study design. But cause-of-death, this is niormally impossible to determine... 10. Anoneuoid on with Lauren Kennedy and Jessica Hullman: "Causal quartets: Different ways to attain the same average treatment effect"February 24, 2023 3:35 PM Mice were typically maintained 5/cage (Supplementary Table S1) and started at 2-5 months of age fed ad libitum (AL) or... 11. Andrew on with Lauren Kennedy and Jessica Hullman: "Causal quartets: Different ways to attain the same average treatment effect"February 24, 2023 3:29 PM Joe: I put it on my website and we submitted it to American Statistician, the journal that published the Anscombe... 12. Peter Dorman on with Lauren Kennedy and Jessica Hullman: "Causal quartets: Different ways to attain the same average treatment effect"February 24, 2023 3:02 PM This is a great post. I went after similar issues about nine years ago in this disquisition: https://econospeak.blogspot.com/ 2014/05/regression-analysis-and-tyranny-of.html My concern... 13. Anoneuoid on with Lauren Kennedy and Jessica Hullman: "Causal quartets: Different ways to attain the same average treatment effect"February 24, 2023 1:28 PM A good example would be survival after a cancer drug. Say the patients normally have expected survival of 2 years,... 14. Joe Nadeau on with Lauren Kennedy and Jessica Hullman: "Causal quartets: Different ways to attain the same average treatment effect"February 24, 2023 1:25 PM Andrew, is this paper published, in press or parked somewhere where it can be cited? Several us are writing genetic... 15. Joe Nadeau on with Lauren Kennedy and Jessica Hullman: "Causal quartets: Different ways to attain the same average treatment effect"February 24, 2023 1:14 PM A great example of focusing on average while ignoring variation (see this paper PMID: 19878144). Background: considerable evidence in humans... 16. Ethan Bolker on David Bowie vs. Beverly Cleary; Dahl advances February 24, 2023 1:00 PM I once rented a house in Southwest Harbor Maine whose two staircases featured in a Beverly Cleary story. Old (or... 17. Tom Passin on How to think about proofs of correctness of computer programs?February 24, 2023 12:44 PM At a different level - and possibly more important for real-world software development - there is Alloy (http://alloytools.org/ about.html), which tries... 18. Blackthorne on with Lauren Kennedy and Jessica Hullman: "Causal quartets: Different ways to attain the same average treatment effect"February 24, 2023 12:37 PM Great paper, I agree that once you start thinking about this sort of thing, it's hard not to think about... 19. Elio on with Lauren Kennedy and Jessica Hullman: "Causal quartets: Different ways to attain the same average treatment effect"February 24, 2023 12:17 PM This is a general effect, of course. Statistical summaries are, of course, summaries, so they cannot provide a full description... 20. the_real_tiddlydump on with Lauren Kennedy and Jessica Hullman: "Causal quartets: Different ways to attain the same average treatment effect"February 24, 2023 11:36 AM I occasionally work with the marketing department within my company in my capacity as a data scientist. Measuring the impact... 21. Christian Hennig on How to think about proofs of correctness of computer programs?February 24, 2023 10:04 AM Not sure how relevant this is, but we have intuition for many mathematical theorems, and often, even if there are... 22. Manuel on David Bowie vs. Beverly Cleary; Dahl advancesFebruary 24, 2023 6:04 AM Bowie. Who else can claim to be the man who fell to Earth AND sold the world? Lots of interesting... 23. Jonathan (another one) on David Bowie vs. Beverly Cleary; Dahl advancesFebruary 24, 2023 12:44 AM I've been listening to Bowie a lot lately. i can give him another few hours for something new. 24. Matt Skaggs on The placebo effect as selection bias?February 23, 2023 3:17 PM That's a great essay, thanks Elio. Very concise. I really liked this summary: "Placebo effects break down into several categories.... 25. njo on How to think about proofs of correctness of computer programs?February 23, 2023 2:44 PM Indeed: https://meatfighter.com/tetromino-computer/ 26. Andrew on How to think about proofs of correctness of computer programs?February 23, 2023 1:23 PM No problem. It's already amazing that a Tetris program can compose blog comments at all! I guess Tetris, like Life,... 27. Tetris on How to think about proofs of correctness of computer programs?February 23, 2023 1:21 PM Im deeply ashamed. Of course I meant Bill when I wrote Mark 28. Tetris on How to think about proofs of correctness of computer programs?February 23, 2023 1:00 PM Mark makes it look like formal program verification is a futile endeavor but that's not completely true! Projects such as... 29. William Gasarch on How to think about proofs of correctness of computer programs?February 23, 2023 12:32 PM (This is Bill Gasarch) Thanks for noticing my post. A few points that are INSPIRED by your post on my... 30. Jonathan Gilligan on Columbia Journalism Review garbles public opinion statsFebruary 23, 2023 12:30 PM FYI, Jeff Gerth has a heck of a history. In the 1990s, his reporting on the nuclear physicist Wen Ho... 31. gg on How to think about proofs of correctness of computer programs?February 23, 2023 11:10 AM I do not know so much about formal program verification. But formal proof verification in mathematics seems to have made... 32. Kaiser on The placebo effect as selection bias?February 23, 2023 10:55 AM I am half way through the causal quartets paper - and I'm so excited this is written down. Have been... 33. Andrew on How to think about proofs of correctness of computer programs?February 23, 2023 10:53 AM Victor: Interesting. This reminds me of the way in which being forced to program an algorithm can reveal gaps in... 34. Victor on How to think about proofs of correctness of computer programs?February 23, 2023 10:49 AM While it is certainly true that Mathematics and Programming are social disciplines one shouldn't ignore the mental discipline that some... 35. Tom Passin on How to think about proofs of correctness of computer programs?February 23, 2023 9:23 AM One problem with formal verification is that the code that is supposed to check the program for correctness will -... 36. Claire in NZ on The Baby Name Voyager is back!February 22, 2023 10:47 PM I'm really late to this party, but I'd think fashion with a genesis in celebrity plays a key role. I'm... 37. Anoneuoid on Manipulating a machine-learning method by feeding it doctored training dataFebruary 22, 2023 8:50 PM But affecting the ordering is nigh impossible to find in retrospect even if you have extremely detailed logs (not that... 38. Nathan on Lady in the MirrorFebruary 22, 2023 8:15 PM >I couldn't figure out where the 2 hour choice is coming from. I would suspect it was a choice made... 39. Nathan on Acupuncture paradox updateFebruary 22, 2023 7:40 PM Yet, there's a strong correlation between nationality and religion, and if somebody was to tell me that nationality was supernatural,... 40. Daniel Lakeland on Manipulating a machine-learning method by feeding it doctored training dataFebruary 22, 2023 5:39 PM Well, Bayesian learning is order invariant if you're using the full system and not taking shortcuts. Also any system that... 41. Andrew on Manipulating a machine-learning method by feeding it doctored training dataFebruary 22, 2023 4:16 PM Phil: Another way to put it is: Yes, you could indeed build an algorithm that would be invariant to the... 42. Phil on Manipulating a machine-learning method by feeding it doctored training dataFebruary 22, 2023 2:58 PM It may be true that if you can change the order that the data are presented then you can also... 43. gwern on Manipulating a machine-learning method by feeding it doctored training dataFebruary 22, 2023 2:58 PM Once you can get in and modify the order of the data, you should be able to change the data... 44. Daniel Lakeland on Manipulating a machine-learning method by feeding it doctored training dataFebruary 22, 2023 2:31 PM This reminds me of sort algorithms. Quicksort has excellent average behavior but n^2 worst case. If you always choose the... 45. Sean on Why I don't trust "libertarian paternalism," part 65 February 22, 2023 12:58 PM Yes, thank you for pointing to the Innocence Project! Just because something is not debated on TV does not mean... 46. Anoneuoid on Software to sow doubts as you meta-analyzeFebruary 22, 2023 12:37 PM Right, rather than a "risk" it is a key part of science that should be celebrated and rewarded. The error... 47. PI on Software to sow doubts as you meta-analyzeFebruary 22, 2023 12:26 PM Never mind - I missed the link at the top! 48. PI on Software to sow doubts as you meta-analyzeFebruary 22, 2023 12:24 PM Is there a write-up on what it does and how it works, other than reading the source code? 49. Matt Skaggs on Software to sow doubts as you meta-analyzeFebruary 22, 2023 10:58 AM "...there's also a risk that some researcher explores a bunch of different potential meta-analytic estimates but only reports one that... 50. Kevin Nelson on Why I don't trust "libertarian paternalism," part 65February 22, 2023 10:37 AM The reason isn't hard to find. Both the number of executions per year and the number of death sentences imposed... Proudly powered by WordPress