[HN Gopher] Not Frequentist Enough
___________________________________________________________________
Not Frequentist Enough
Author : luu
Score : 72 points
Date : 2022-10-06 21:35 UTC (1 days ago)
(HTM) web link (statmodeling.stat.columbia.edu)
(TXT) w3m dump (statmodeling.stat.columbia.edu)
| konschubert wrote:
| When studying statistics, I never understood the apparent
| conflict between frequentism and bayesianism.
|
| To me, they seemed like complementary tools, each with their own
| strengths and weaknesses.
|
| You can use one or the other depending on what your goal is. Do
| you want to figure out how much you should believe something?
|
| Or are you trying to figure out how compatible a hypothesis is
| with reality? In the first case, go bayesian, in the second,
| frequentist.
| rwilson4 wrote:
| There is less of a conflict than many would have you believe.
| In many situations, both approaches yield the same answer.
| There are some edge cases. For example, in A/B testing, is
| early peeking bad? From a frequentist perspective the answer is
| "yes, either use a sequential method, or don't early peek at
| all". From a Bayesian perspective the answer is "early peeking
| is fine".
|
| It boils down to what properties you want your analysis to
| have. Cox and Hinkley's "Theoretical Statistics" has a great
| discussion (section 2.4). Basically, you might want your
| analysis to have a certain kind of internal consistency. But
| you might also want your analysis to be replicable either by
| yourself or by another researcher. Those both seem like pretty
| important things! But there are edge cases (like the early
| peeking example) where you can't have it both ways. So you have
| to pick which one you want, and use the corresponding methods.
| zozbot234 wrote:
| The likelihood principle actually supports the Bayesian
| perspective on these issues of experiment design, and is
| regarded as foundational by many frequentists.
| rwilson4 wrote:
| Agreed. But as Cox and Hinkley discuss, the likelihood
| principle is sometimes at odds with the repeated sampling
| principle, so in any particular application, you need to
| identify if there is a conflict, and if so, which principle
| is more important. In my domain (simple A/B tests), you can
| claw the repeated sampling principle from my cold, dead
| hands.
| hackandthink wrote:
| If you want to know: do Higgs Bosons exist, you can go
| frequentist or bayesian:
|
| Wasserman: p-value is fine for Higgs experiment:
|
| https://normaldeviate.wordpress.com/2012/07/11/the-higgs-bos...
|
| Tom Campbell-Ricketts: Bayes is better even for Higgs
| experiment
|
| https://maximum-entropy-blog.blogspot.com/2012/07/higgs-boso...
|
| My take: frequentist p-value is simpler but Bayes is what you
| really want.
| kgwgk wrote:
| Nothing prevents you from "going Bayesian" in figuring out how
| compatible a hypothesis is with reality. Bayesians have no
| issues with probabilities representing frequencies even though
| frequentists cannot understand probabilities representing
| uncertainty.
| jxy wrote:
| Seriously? Frequentists simply build infinite amount of
| parallel Universes and ask what percentage of the Universes
| It occurs with certainty.
| kgwgk wrote:
| What's the probability that Russian drones attacked the
| Nord Stream pipeline?
|
| It's the fraction of parallel universes where that happened
| - or something.
| layer8 wrote:
| That sounds like a pretty sensible approach if you believe
| in Many-Worlds.
| konschubert wrote:
| It has nothing to do with that. Even if there is a single
| world, it is enough to make the thought experiment.
| juped wrote:
| I think the "conflict" is basically just a matter of Jaynes
| having used the word "frequentist" like a hardcore Calvinist
| uses the word "Arminian".
| pdonis wrote:
| On the view of at least some Bayesians, "frequentist" is just
| the special case of "Bayesian" that you get when you are
| computing credences based on a large number of identically
| prepared, independent trials over a known sample space. So on
| this view (with which I tend to agree), the two are certainly
| not incompatible.
|
| It is of course possible to do both frequentist and Bayesian
| statistics badly. I would say bad frequentism comes when one
| fails to realize that standard frequentist methods tell you the
| probability of the data given a hypothesis, when what you
| really need to know is the probability of the hypothesis given
| the data. Bayesianism at least starts right out with the latter
| approach, so it avoids the former (unforfunately all too
| common) error.
|
| Bad Bayesianism, OTOH, I would say comes when one fails to
| realize that Bayes' rule is not a drop-in replacement for your
| brain. You still need to exercise judgment and common sense,
| and you still need to make an honest evaluation of the
| information you have. You can't just blindly plug numbers into
| Bayes' rule and expect to get useful answers.
| grayclhn wrote:
| IME "bad Bayesian analysis" is when people use plausibly
| defensible priors to manufacture outcomes... which happens
| all the fucking time.
| analog31 wrote:
| Long ago when I was a physics grad student, I distinctly
| remember that when someone introduced Bayesian statistics
| in a talk, it was because they were trying to justify
| weeding outliers from their data by hand. And they always
| got called to task on it.
| BeetleB wrote:
| > When studying statistics, I never understood the apparent
| conflict between frequentism and bayesianism.
|
| I remember a statistician once saying: There are two types of
| statisticians: Those that are Bayesian and those that are both
| Bayesian and frequentist.
| hackandthink wrote:
| A famous Bayesian arguing for frequentist statistics?
|
| Gelman tries to steal the concept "Frequentism" from simple
| minded frequentist statisticians.
|
| His argument seems to be:
|
| Simple minded frequentists statisticians perform a statistical
| procedure once - they do not think about performing the procedure
| many times.
|
| They fall into this trap (from Gelman's paper):
|
| "3. Researcher degrees of freedom without fishing: computing a
| single test based on the data, but in an environment where a
| different test would have been performed given different data"
| AstralStorm wrote:
| The frequentist test for this attempts to see what would happen
| with a variety of test designs using likelihood ratio and
| similar statistical tests. Relaxing it you end up with
| Generalized Method of Moments family.
|
| A Bayesian would attempt to compute the Bayes factor using
| approximate Bayesian computation resulting in more or less the
| same thing. You end up with various information criteria.
|
| Both approaches then converge in using Monte Carlo techniques
| to evaluate the features of the whole experimental setup using
| simulated data.
|
| All of the above approaches replace the problem of choosing the
| test/design based on data by the researcher with one by a data
| driven algorithm with known properties.
| rwilson4 wrote:
| Gelman is one of the few self-proclaimed Bayesians who doesn't
| seem to outright hate frequentist approaches. They're
| complementary approaches. Bayesian methods are great for
| combining different sources of information. Frequentist methods
| are great for validating that a method is working well. (For
| example, Gelman often recommends running simulations to see if
| models give sensible predictions, but that is itself a pretty
| frequentist thing to do.)
|
| Frequentism is mostly about how to _evaluate_ a methodology. It
| 's pretty agnostic about what that methodology is. Bayesian
| methods are about combining different sources of information.
| In a situation where you only have one source of information,
| Bayesian and Frequentist methods usually give the same answer.
|
| People say you might as well always use Bayesian methods then.
| But no matter what, you should always try to validate or poke
| holes in your model, and Frequentist techniques are great for
| that. So it's best to be familiar with both!
| hackandthink wrote:
| yes
|
| https://stats.stackexchange.com/questions/115157/what-are-
| po...
| kgwgk wrote:
| > running simulations to see if models give sensible
| predictions, but that is itself a pretty frequentist thing to
| do
|
| Is looking at probability distributions "a pretty frequentist
| thing to do"? Even when those models and simulations include
| _prior_ probability distributions? Sure, one can (re)define
| frequentist to include Bayesian models - as Gelman seems to
| want to do in that post. I just don't see how this helps to
| clarify anything.
| youainti wrote:
| My understanding of the post is that given a small actual effect
| size, for a fixed experiment, you are more likely to get a
| significant p-value on a "large" measured effect size.
| tpoacher wrote:
| There's nothing stopping the definition of a p-value from
| incorporating bayesian priors.
|
| There. I said it.
| hammock wrote:
| A lot of words to say something that essentially boils down to,
| "Make sure your findings replicate."
| hackerlight wrote:
| More like "make sure your test power is what you think it is".
| There will still be results that fail to replicate by virtue of
| the rejection of the null hypothesis by chance, but that should
| only happen 1 in 20 times at an alpha of 0.05. With all the bad
| practices that alter test power, such as p-hacking and the file
| drawer effect, that 1 in 20 blows up to 1 in 2.
| hackandthink wrote:
| Gelman makes a distinction between p-hacking and choosing the
| test based on your data.
|
| p-hacking is confirming your preferred hypothesis - just try
| again.
|
| the latter let's you write a paper.
|
| (test power is different: if there's really an effect, would
| you detect it)
| yellowstuff wrote:
| Most people care about their findings replicating. The hard
| part is knowing what you need to do to improve your chances.
___________________________________________________________________
(page generated 2022-10-07 23:00 UTC)