[HN Gopher] Objective Bayesian Hypothesis Testing
___________________________________________________________________
Objective Bayesian Hypothesis Testing
Author : rnburn
Score : 71 points
Date : 2024-08-19 18:04 UTC (4 days ago)
(HTM) web link (www.objectivebayesian.com)
(TXT) w3m dump (www.objectivebayesian.com)
| elmomle wrote:
| When the author says "objective" they are referring to a prior
| that gives equal weight to values within the null hypothesis and
| to those without (along with a few other things: symmetric and
| non-increasing away from the mean). I appreciate this approach,
| and think there's much to commend it, but think that that's a key
| thing to be aware of (because any use of "objective" when
| referring to priors is, shall we say, dubious).
| vcdimension wrote:
| Yes, it would be nice to know how things change for different
| weightings of the null and alternative priors.
| vcdimension wrote:
| This article is very interesting and informative, however it's a
| bit ironic that an article about misinterpretations of the
| meaning of the p-value, misinterprets the misinterpretation; in
| the first blue box it's clear that Bernstein is interpreting the
| p-value as the probability of randomly rejecting the null (which
| is what you do when you get something statistically significant)
| yet in the text following that they say he's interpreting it as
| the probability of the null. Bernsteins mistake is that he
| appears to interpret it as an unconditional probability rather
| than a conditional one (correct interpretation; p-value =
| Prob(rejecting the null when the null is true)).
| null08 wrote:
| Yes I had the same issue. But the wording "there is a < 5%
| probability that an outcome was the result of chance" is in
| fact problematic since many readers will go on to conclude
| "hence a >95% probability that the outcome was not the result
| of chance", so it is easier to misinterpret than the technical
| definition P( Observation | H_0 ).
|
| In courses I will typically use wordings like "If there was
| truly no association, then the probability of getting an
| observation like this is <5%".
| kqr wrote:
| > correct interpretation; p-value = Prob(rejecting the null
| when the null is true)
|
| This is also not quite correct. The p-value is the probability
| of falsely rejecting the null _due to sampling error_. It is
| quiet on all other errors that are frequently committed.
|
| The real probability of falsely rejecting the null starts at 15
| % thanks to mathematical slip-ups alone: https://two-
| wrongs.com/the-lying-p-value
| nalzok wrote:
| > by kqr, published 2024-11-19
|
| It's from the future! ;)
| underlines wrote:
| A great question that I came across in Hypothesis Driven
| Development a long time ago: Should you use Frequentist
| Statistics or Bayesian Statistics? It's relevant when you do A/B
| or Multivariate Testing.
|
| As it was very difficult for someone like me without higher stats
| or math education, I can highly recommend the following
| additional sources:
|
| - https://www.redjournal.org/article/S0360-3016(21)03256-9/ful...
|
| - https://amplitude.com/blog/frequentist-vs-bayesian-statistic...
|
| - https://indico.cern.ch/event/568904/contributions/2651065/at...
| shiandow wrote:
| The Bayesian approach to A/B testing gives an interesting
| example of how frequentists and Bayesian approaches can differ.
|
| A frequentist approach tries to limit the probability that a
| test setup will accept a 'false' result, one that could simply
| arise by chance.
|
| A Bayesian approach actually calculates the probability that a
| test result could occur 'by chance'. You can then stop the test
| at any point and be sure you only accept <x% of results that
| could occur by chance, by the power of expectation values you
| never breach the x% limit no matter how often you 'stop' the
| test.
|
| The interesting thing is that while these would seem to be very
| similar, there actually isn't anything stopping the Bayesian
| approach from accepting _any_ test _eventually_. Giving it 0
| statistical power in the frequentist sense. The only thing the
| Bayesian approach ensures is that for any 'false' test you
| accept after time T there are many more that will keep running.
| Vecr wrote:
| Why would you care about that though? Calculate the odds
| between your hypotheses, not the probability you'd ever see
| one.
| shiandow wrote:
| The Bayesian stance is that you should not care. The
| frequentist stance is that a test that has a p-value of 1
| is the worst possible.
|
| My stance is that you should know why to care about either.
| Oh and that the thing you're calculating an expected value
| off should somehow contribute _linearly_ to your profits
| /costs, averages do strange things to nonlinear functions.
| LegionMammal978 wrote:
| Eh, even an expected value that's linear with respect to
| profits can end up with strange results like the St.
| Petersburg paradox. In general, naively maximizing it
| breaks down at the point where you stop being insensitive
| to the possible risks.
| bookofjoe wrote:
| Off topic but topical: Mike Lynch's yacht was named "Bayesian"
| vcdimension wrote:
| So I guess we'll never know the p-value of that event...
___________________________________________________________________
(page generated 2024-08-23 23:02 UTC)