[HN Gopher] Bayesians moving from defense to offense
___________________________________________________________________
Bayesians moving from defense to offense
Author : Tomte
Score : 131 points
Date : 2023-12-23 14:48 UTC (8 hours ago)
(HTM) web link (statmodeling.stat.columbia.edu)
(TXT) w3m dump (statmodeling.stat.columbia.edu)
| tomrod wrote:
| Huh. Wouldn't publication bias dramatically impact a Bayesian's
| conclusions?
| ta988 wrote:
| Depends how you model it. I would say not taking into account
| the publication bias if you have ways to measure or at least
| estimate its distribution is a bigger mistake to make.
| ChadNauseam wrote:
| Yes, it should affect anyone's conclusions unless they know
| exactly how much publication bias there is and then account for
| it
| pacbard wrote:
| Yes, possibly. Publication bias is a known problem when using
| prior work (see [1] for some references on the problem and
| potential solutions).
|
| One of the strengths of Bayesian methods is that you can use a
| less informative prior to model your skepticism of prior work.
|
| [1]: https://www.sciencedirect.com/topics/medicine-and-
| dentistry/....
| epgui wrote:
| Yes, but no more than a non-Bayesian's. The difference is
| mainly that the Bayesian approach makes more of its assumptions
| explicit.
| jvans wrote:
| This. Bayesian assumptions are explicit and you'll be forced
| to defend your priors. Frequentist assumptions like
| homoscedasticity are implicit and rarely challenged.
|
| I wouldn't be surprised if many people using frequentist
| methods have no idea what assumptions need to be true for the
| methods to be valid. In a Bayesian framework you can't avoid
| the assumptions
| ajkjk wrote:
| That's why it's for constructing a prior that you change later;
| your study itself should update the prior in the direction of
| being more correct. But it's not like you can do better than
| "the best available prior" (by definition).
| mjfl wrote:
| I've never heard anyone promote frequentist statistics. I've only
| ever heard Bayesians on the 'offense' talking about why their
| framework is better. To be clear I am someone who doesn't really
| understand the faultlines.
| analog31 wrote:
| Indeed, when I took statistics in college, nearly 40 years ago,
| we proved Bayes theorem, and worked through several
| applications, so it could hardly have been regarded as
| controversial.
| adastra22 wrote:
| It's not. Statistics is statistics. This controversy is new
| and contrived.
| fbdab103 wrote:
| Tell that to Ronald Fisher, who made it a point to deride
| Bayesian stats wherever there was an opportunity.
| jvans wrote:
| The controversy is decades old and largely resolved. Both
| methods work well on large enough data sets, Bayesian
| methods are far superior on small datasets
| adastra22 wrote:
| This statement doesn't really make sense to me. A
| frequentist approach assumes a random (or noisy)
| underlying process and characterizes its probability. A
| Bayesian approach assumes a fixed but unknown process and
| updates the probability a given model matches that
| reality.
|
| It is ultimately the same math, if you are comparing
| apples to apples. Just different ways of looking at a
| problem. Sometimes one is better suited than the other.
|
| It's like as if physicists were arguing over whether
| Cartesian or Polar coordinates were better. It's the same
| damn physics, just expressed differently. In some
| problems one approach is easier to work with than the
| other, and can even make seemingly intractable problems
| solvable. But that doesn't mean the other approach was
| "wrong."
| tnecniv wrote:
| Bayes theorem is fairly elementary probability. It isn't the
| part that is questioned in Bayesian statistics. Someone
| disagreeing with Bayes theorem would be like disagreeing with
| addition.
|
| The controversial part is the methodology for statistical
| analysis built on top of it (e.g., you need a prior but where
| did that come from?)
| Tomte wrote:
| Bayes theorem shares a name with Bayesianism, but it is not
| specific to it. No frequentist would question Bayes theorem,
| it is on the level of 1+1=2.
| SpaceManNabs wrote:
| > I've never heard anyone promote frequentist statistics
|
| I will do it:
|
| 1. Computationally easier
|
| 2. often analytical theory available for most use cases so
| interpretability is high
|
| 3. more literature available so you can get unstuck faster if
| you mess up
|
| 4. no accusations of subjective bias in your prior (the con is
| clear, no ability to leverage subjective expertise)
|
| 5. In the asymptotic regime, MLE and bayesian MAP often
| converge anyways
|
| 6. king of hypothesis testing
|
| For most people, it doesn't matter. It matters when you are
| doing treatment for small sample sizes or other situations that
| would cause low power.
| bogtog wrote:
| > king of hypothesis testing
|
| This advantage can't be understated. Researchers, at least in
| psychology, almost never seem to care about the magnitude of
| an effect. It's simply enough to show that some effect
| happens in some direction. For this (generally acceptable)
| purpose, frequentist stats are great.
| urschrei wrote:
| A statistically significant result without something like a
| Hedges G (or Cohens D if comparing means) measure would be
| called out by even a moderately attentive reviewer as a
| matter of course (especially if small sample sizes are
| involved), and many reputable journals actually require
| effect sizes to be stated, so you run the risk of being
| desk rejected for leaving it out.
| travisjungroth wrote:
| Add some sampling bias, a significance threshold of 0.05.
| Baby, you've got a replication crisis going!
| zozbot234 wrote:
| Frequentist statistics is largely based on the maximum
| likelihood principle, which is a proper Bayesian analysis
| whenever the prior is a flat distribution. And since inference
| is mostly done on model _parameters_ in frequentist statistics,
| the distribution you assume for them will also depend on how
| you parameterize the model. So it can often be justified as a
| convenient approximation to a "properly" Bayesian result.
| jdewerd wrote:
| I've probably heard 1000 Bayesians rant about the alleged
| Frequentist consensus and 0 Frequentists complain about
| Bayesian analysis. I'd need some pretty steep priors to
| interpret this as anything other than the academic equivalent
| of "one WEIRD trick THEY don't want you to know" marketing.
| paulsutter wrote:
| If we were allowed one super-upvote per month, I would use
| mine right now on your comment
| selimthegrim wrote:
| Dang is going to file this under the Bumble model of HN
| monetization in his desk drawer
| causality0 wrote:
| There are times I'd be willing to pay a fiver to un-flag
| my comment.
| jowea wrote:
| Or just HN Gold
| throwawaymaths wrote:
| There's always room for a first time.
|
| Bayesian statistics is sound but I suspect it's often just
| used to justify biases. It is technically valid to use a
| prior and _de facto_ never update it, because you know I 'll
| get around to updating my prior next week, or... eventually,
| cough cough _let 's be honest, never_
| marcosdumay wrote:
| Bayesian statistics is the one where you must state your
| bias explicitly, and justify them one by one under the
| light.
|
| It's the frequentist one that gets your biases implicitly,
| on the form of corrections and hypothesis formulation, so
| that people don't notice them.
| exe34 wrote:
| Every single statistics class I've ever taken or been sent to
| has been 95% frequentist with a rushed Bayesian digression
| near the end. I've submitted papers with Bayesian work and
| the reviewer asked for p-values and would not budge. I gave
| him his bloody p-values, because I could not afford not to
| get the paper published that early in my career.
| jdewerd wrote:
| There's always a more complicated model.
| zozbot234 wrote:
| The "more complicated" version of frequentist statistics
| is called robust statistics. There's most likely a way to
| rephrase your favorite Bayesian analysis so as to make it
| fully kosher from a "robust+frequentist" point of view,
| even keeping the math unchanged. It just goes to show how
| silly the "controversy" is.
| PheonixPharts wrote:
| Bayesian statistics is fundamentally _less_ complicated
| than Frequentist statistics since everything can be
| derived from a very simple set of first principles,
| rather than complex frameworks of ad hoc testing
| methodologies.
| light_hue_1 wrote:
| Yes, because Frequentists won a long time ago. And are
| responsible for producing garbage science ever since.
|
| Of course the winners don't rant about anything. But any time
| we probe the consequences of Frequentist statistics they turn
| out to be horrific for science, our health, and our planet.
| travisjungroth wrote:
| Frequentist is the default. People don't promote it. They just
| teach it _as_ statistics, use p-values, and look funny at
| anyone who does otherwise.
|
| It's like how you wouldn't hear people in the US promote
| imperial measurements for home baking. When you dominate, you
| don't have to advocate.
| analog31 wrote:
| I'd identify a larger problem, which is teaching anything as
| a bunch of recipes.
|
| My college offered stats in two tracks: There was a one-
| semester course for science majors, which was mostly plugging
| numbers into formulas. The course was utterly baffling for
| most of the students who took it.
|
| There was also a two-semester course for math majors. It was
| mainly about proofs, but also had time to go into more depth.
| You have to know the assumptions underlying a formula, if
| you're expected to prove it. ;-) But it was only taken by the
| math majors.
| mjfl wrote:
| But I don't see p-values as connected to frequentist
| statistics. Rather, p-values are a concept that applies to
| the _problem_ of hypothesis testing, which both frequentist
| and bayesian statistics should apply to, right? And in both
| cases you would use a p-value. The unique thing that Bayesian
| statistics can do is fit parameters can calculate uncertainty
| in those parameters, which does not contradict any of the
| concepts of hypothesis testing.
| travisjungroth wrote:
| A p-value is the frequency you'd expect to see a sample
| with an effect this size or greater, given the null
| hypothesis. It's inherently a frequentist approach to
| inference.
|
| You can squish it into Bayes by considering it a uniform
| prior on the real number line that you never update, but
| you're not really doing things "in the spirit" of Bayes,
| then.
| mjfl wrote:
| What is a "prior" in the context of hypothesis testing?
| That seems to me to be a concept used in parameter
| estimation, with Bayes rule. But hypothesis testing is
| not parameter estimation.
| travisjungroth wrote:
| As the linked post points out, you can regularize a
| p-value based on prior experiments.
|
| IMO, null hypothesis testing is way overused. We should
| be quantifying the comparison of the null versus
| alternative hypotheses. People already compare them.
| Might as well bring some math into it.
| mitthrowaway2 wrote:
| How is hypothesis testing different from parameter
| estimation? They seem conceptually the same to me, with
| one a special case of the other.
| PheonixPharts wrote:
| In Bayesian terms there is no distinction between
| hypothesis testing and parameter estimation.
|
| If your hypothesis is that A is greater than B, then
| you're test boils down to "The probability that A is
| greater than B", which is arrived at through parameter
| estimation via Bayes' Theorem.
|
| You can consider the distribution of estimates for a
| parameter to represent a space of possible hypothesis for
| the true value of that parameter and their absolutely or
| relative likelihood based on the information available.
|
| The also has the pleasant consequence that hypothesis
| testing using Bayesian methods nearly always is closer to
| the actual question someone wants to answer rather than
| replying with confusing statements such as "the
| probability of rejecting the null hypothesis", which
| contains an implicit double negative.
| tingletech wrote:
| I think with a Bayesian test one would use credible
| intervals rather than p values?
| PheonixPharts wrote:
| > But I don't see p-values as connected to frequentist
| statistics.
|
| P-values are fundamentally Frequentist because this
| framework see that _observed data_ as random, whereas
| Bayesian statistics believe our observations are the only
| thing that is actually known, and all the other parameters
| are what is random in our experiments. That is: the data is
| the only part of an experiment that is real, everything
| else that we can 't directly observe is something we must
| hypothesize about.
| mapmeld wrote:
| I went to a week-long Probabilistic AI seminar, and I got the
| impression there was an era when enough stats and math
| professors did not take a Bayesian framework seriously, that
| there was a risk someone's thesis or dissertation defense would
| be derailed by someone on faculty.
| Avshalom wrote:
| It's worth pointing out that the vast majority of "Bayesians"
| you see talking shit have no actual training in bayesian
| statistics, or any statistics training beyond the college-intro
| level.
| hackandthink wrote:
| Link to the first quoted paper:
|
| http://www.stat.columbia.edu/~gelman/research/published/pval...
| pil0u wrote:
| What resources would you recommend to implement an A/B test
| evaluation framework using a Bayesian approach?
|
| I would love to embrace (or at least try to) such a new approach,
| but it feels like without a PhD in stats it's hard to get it
| started.
| jvans wrote:
| Statistical rethinking by McElreath is an exceptional resource
| for Bayesian modeling. It's very accessible and gives you some
| good examples to work through.
|
| Some very basic Bayesian models can go a long way towards
| making informed decisions for a/b tests
| plants wrote:
| Specifically for A/B or A/B/N testing, you can use a beta-
| bernoulli bandits, which give you confidence about which
| experience is best and will converge to an optimal experience
| faster than your standard hypothesis test. Challenges are that
| you have to frequently recompute which experience is best and
| thus, dynamically reallocate your traffic. They also only works
| on a single metric, so if your overall evaluation criterion
| isn't just something like "clickthrough rate", this type of
| testing becomes more difficult (if anyone else knows how
| multiple competing metrics are optimized with bandits, feel
| free to chime in).
| HuShifang wrote:
| Here's a recent webinar from PyMC Labs' Max Kochurov (with some
| slides and code too):
|
| https://www.youtube.com/watch?v=QllfKQH-yQ4
| fifilura wrote:
| I am not the expert and humbled by all the experts in this
| thread.
|
| But I wanted to say that when I looked A/B testing few years
| ago I started off with this book.
|
| https://github.com/CamDavidsonPilon/Probabilistic-Programmin...
|
| And somehere down the line he will introduce the Beta/Binomial
| method for A/B testing.
|
| In my (again humble) understanding the benefit of doing this in
| the Bayesian way is both that you can actually get an
| understandable answer, but also that the answer does not have
| to be the final result you can continue by adding a loss
| function for example.
| esafak wrote:
| It really is not; Bayesian statistics is way more intuitive.
| For a practical guide see
| https://dataorigami.net/Probabilistic-Programming-and-Bayesi...
|
| For Bayesian ML see https://probml.github.io/pml-
| book/book1.html and https://www.bishopbook.com/
|
| For Bayesian statistics see
| http://www.stat.columbia.edu/~gelman/book/
| tmoertel wrote:
| One straightforward approach is "Test & Roll: Profit-Maximizing
| A/B Tests" https://arxiv.org/abs/1811.00457
| clircle wrote:
| I don't really see Frequentism and Bayes to be in conflict. You
| use them to answer different questions. I wouldn't test a point
| null hypothesis with a Bayesian model.
| kneel wrote:
| It honestly just seems like an abstract debate to claim that
| your logical framework is somehow superior. Different tools for
| different datasets
| mitthrowaway2 wrote:
| I think the Bayesians argue that you _would_ , so there does
| seem to be a conflict.
| zestyping wrote:
| I don't know when Bayesians have ever been on defense. They've
| always been on offense.
| paulsutter wrote:
| Bad reasoning trumps arcane methodology debates every time
___________________________________________________________________
(page generated 2023-12-23 23:00 UTC)