[HN Gopher] Well-known paradox of R-squared is still buggin me
___________________________________________________________________
Well-known paradox of R-squared is still buggin me
Author : luu
Score : 62 points
Date : 2024-06-30 08:09 UTC (4 days ago)
(HTM) web link (statmodeling.stat.columbia.edu)
(TXT) w3m dump (statmodeling.stat.columbia.edu)
| lmm wrote:
| > The two states are much different in their politics!
|
| Are they? Sounds like they're both swing states, pretty close to
| 50-50, so which state you're from doesn't have a big effect on
| what your politics are likely to be. Which is exactly what the
| R^2 tells us. Where's the paradox?
| derbOac wrote:
| That was my first thought as well.
|
| Also, traditional R squared with binary variables, or maybe
| categorical variables, never made much sense to me. The
| "meaning" of the variance I don't think is quite the same. You
| generally have to nonlinearly transform (e.g., logit) any
| linear model quantities into something else to put it on an
| observed variable scale.
| spenczar5 wrote:
| A ten point gap is not a swing state, that's big. Its about the
| same as New Jersey vs Texas in the 2020 presidential vote.
| zarzavat wrote:
| In the US.
|
| In other countries 45/55 could easily be a swing state.
|
| For example the 2014 Scottish independence referendum was
| decided 45/55 and that is usually held to be a fairly close
| result, at least close enough that it hasn't ended the
| question.
|
| So the real story here is voter polarization. The reason that
| Texas isn't a swing state is because there's a big hard core
| of immoveable voters at either end. So the actual population
| of swing voters is small.
| chipdart wrote:
| > A ten point gap is not a swing state, that's big.
|
| No, it isn't. Your "ten point gap" is something like a 5%
| delta. That's below the confidence level of some polls.
| DougBTX wrote:
| > Where's the paradox?
|
| Exactly. I propose that the paradox is in first-past-the-post
| voting, a 5% swing leads to a 100% change in representation.
| How can that be?
| gadders wrote:
| Relax - they're all broken:
| https://www.investopedia.com/terms/a/arrows-impossibility-
| th...
| evan_piermont wrote:
| Only if you reject the axiom of choice:
| https://www.jyotirmoy.net/posts/2013-10-28-arrow-
| ultrafilter...
| lupire wrote:
| Only if you have infinite voters.
| xigoi wrote:
| Arrow's impossibility theorem only applies to ranked choice
| voting systems.
| cubefox wrote:
| ... which are almost never used in voting for
| governments.
| lupire wrote:
| FPTP is a ranked choice system.
| gadders wrote:
| Yes, but what I mean is that even if you move away from
| FPTP voting, the others all have compromises.
| isotypic wrote:
| But for many the drawbacks of ranked choice systems are
| far more preferable to those of FPTP. Additionally,
| Arrow's theorem only states that the spoiler effect
| cannot be completely eliminated by ranked choice - it
| says nothing about how often such an event actually
| occurs, and in practice spoiler candidates will occur
| less frequently with ranked choice systems compared to
| FPTP.
|
| Additionally, rated choice voting systems are not subject
| to Arrow's theorem.
| ninjinxo wrote:
| If voters are split 60-40 on an issue, that doesn't mean that the
| odds are 60-40.
|
| You should instead be asking, what are the odds that that X
| voters could change their vote.
| akira2501 wrote:
| What paradox? People don't vote a particular way because they
| live in a state. The logic here would imply "Welp, I live in
| Kentucky, so I guess Red?" would be the expected mode at the
| voting booth.
| jncfhnb wrote:
| Statistics does not require asserting causality.
| akira2501 wrote:
| Yes but you do actually have to understand the outcome you're
| measuring for.
| jncfhnb wrote:
| Not to encounter the "paradox" of R2 being low while a real
| effect is identified
| cperciva wrote:
| I don't see the problem. The R-squared is 0.01 for blue/red
| predicting individual votes, because both of the states in
| question are really just different shades of purple. The
| R-squared is 1.00 for predicting the total vote share and which
| party wins the state, because of course the red/blue binary
| completely determines those.
| energy123 wrote:
| R-squared is very small when the effect is small because it's
| squared, as the name implies.
|
| If he doesn't like that he should just use R by itself, which
| would turn that 0.01 into 0.1, and it'd turn that 0.16 into
| 0.4. R is the Pearson correlation coefficient of a univariable
| linear regression.
| cubefox wrote:
| (Who came up with R squared anyway? Seems like someone wanted
| to remove the possible minus sign from R but didn't know
| about the ABS function, so he used the square which has this
| side effect of making the numbers too small.)
| carlob wrote:
| I don't think it's about not knowing the abs function, more
| about the fact that the first derivative would be
| discontinuous and the second doesn't exist in 0. Variance
| had much nicer properties mathematically than absolute
| deviation.
| mattalex wrote:
| You can solve L1 regression using linear programming at
| fantastically large scales. In fact in many applications
| you do the opposite: go from squared to absolute because
| the latter fits into in lp
| snarkconjecture wrote:
| R squared has extremely nice properties. If you add a bunch
| of linearly uncorrelated variables together, then compute
| R2 between each variable and the sum, the R2 values will
| sum to 1.
| cubefox wrote:
| R (r) has also a nice property:
|
| > if X,Y,Z are "generic" random variables such that
| r(X,Y)=a and r(Y,Z)=b, we should on average expect that
| r(X,Z)=ab.
|
| https://www.lesswrong.com/posts/vfb5Seaaqzk5kzChb/when-
| is-co...
| justk wrote:
| The reason of the apparent paradox: a) in this case the
| model is a mixed model b) second the variable are nominal
| so you have to select one of the pseudo R^2 models. For
| more information: (1) Pseudo R-squared:
| https://en.wikipedia.org/wiki/Pseudo-R-squared (2) R
| squared for mixed models - the easy way https://ecologyfora
| crowdedplanet.wordpress.com/2013/08/27/r-...
|
| c) The R^2 used with a linear model requires a constant
| term, in this case the constant term or bias explains a lot
| about preferences (almost 50/50) so there is less
| information available for the slope term.
|
| Hope this helps.
| vcdimension wrote:
| @justk What you're talking about might make sense if
| there were more independent variables to consider, but in
| this case there's only one, state. So in fact you could
| say that there are two conditional linear models in the
| example; one for the first state (state=0), and one for
| the second (state=1). The model does the best job with
| the information available (state).
| justk wrote:
| Sorry, I edited my post several times and finally choose
| a short form with links other sources. If you fix state=1
| then there are no more random variables so the R^2
| doesn't have any meaning. Just for fun, what the model
| should predict for state = 0.5?, that corresponds to a
| person that is 50% in the red state and 50% in the blue
| state, I think a mixed model is appropriated here when
| the state variable is discrete, so that each value of the
| state variable represents a different part of the
| population, the other model should be used when people
| move a lot and change frequently the state where they
| vote in, but in that case you should have to consider the
| fluctuations in the total population in each state at the
| time of voting.
| vcdimension wrote:
| @justk The R^2 value of 0.01 calculated on that webpage
| uses both states, not just one: the variance of the
| predicted values across both states is 0.55^3+0.45^3 -
| (0.55^2+0.45^2) [?] 0.497 [?] 0.5 I don't think it makes
| sense to use a mixed model in this case since the
| variance is the same for each state. A mixed model is
| used when the observations have some structured
| heteroskedasticity, i.e. different variances for
| different values of the independent variables.
| fjkdlsjflkds wrote:
| > The R^2 used with a linear model requires a constant
| term, in this case the constant term or bias explains a
| lot about preferences (almost 50/50) so there is less
| information available for the slope term.
|
| This explains the paradox, basically. When you take the
| null model "preference = 50%" (i.e. intercept-only
| model), there simply isn't much residual variance left
| for the linear model to explain.
|
| That's why you get an R^2 = 1 if you use the "R^2 =
| rho(state, preference)^2" formula (you are ignoring the
| role of the intercept in explaining most of the variance,
| and exploiting the translation-invariance of the Pearson
| correlation) vs. you getting an R^2 = 0.01 when you use
| the (more correct) "R^2 = explained variance / total
| variance" formula.
|
| TL;DR: It makes sense to get a very low R^2 when it is
| the _intercept_ and _not the predictor_ that is
| explaining most of the variance.
| kgwgk wrote:
| > there simply isn't much residual variance left for the
| linear model to explain.
|
| I'd say that there is still quite a lot of residual
| variance to explain. You need a baseline - the worst
| choice would be to predict 0 (or 1) and the mean squared
| error would be 0.5. Using 0.5 as baseline halves the mean
| squared error to 0.25.
| fjkdlsjflkds wrote:
| This, of course, will depend on how you code your
| variables, but if you try to fit a null, intercept-only
| and predictor-only model, you get this as residual
| variance:
|
| > data <- data.frame(state = c(0, 1), pref = c(0.45,
| 0.55))
|
| > sum(residuals(lm(pref ~ 0, data = data))^2) # null
| model
|
| [1] 0.505
|
| > sum(residuals(lm(pref ~ 1, data = data))^2) #
| intercept-only model
|
| [1] 0.005
|
| > sum(residuals(lm(pref ~ state + 0, data = data))^2) #
| predictor-only model
|
| [1] 0.2025
|
| So, it seems clear that you only get a "perfect"
| prediction with the full (intercept + predictor) model
| mostly because of the intercept (which explains
| (0.505-0.005)/0.505 = 0.99 = 99% of the variance).
|
| Thus, it makes sense that the predictor is only
| explaining the rest (i.e. 1%) of the variance... hence,
| the R^2 = 0.01
| kgwgk wrote:
| Your calculation is not directly related to the model
| (and associated R2) discussed in the article which are
| about the prediction of individual votes using the state
| as predictor - not state averages using the state as
| predictor.
|
| Maybe I'm completely missing your point but the
| calculations in the blog post are, adapting your code (I
| think you meant mean where you wrote sum):
| > data <- data.frame(state = rep(c(0, 1), each=20), pref
| = c(rep(0, 11), rep(1, 9), rep(0, 9), rep(1, 11)))
| > mean(residuals(lm(pref ~ 0, data = data))^2) # null
| model [NOT IN THE BLOG POST] [1] 0.5 >
| mean(residuals(lm(pref ~ 1, data = data))^2) # BASELINE
| intercept-only model [1] 0.25 >
| mean(residuals(lm(pref ~ state + 0, data = data))^2) #
| predictor-only model [NOT IN THE BLOG POST] [1]
| 0.34875 > mean(residuals(lm(pref ~ state, data
| = data))^2) # MODEL [1] 0.2475 >
| summary(lm(pref ~ state, data = data))$r.squared # MODEL
| 0.01
|
| The blog post is about what you call "intercept-only"
| model (MSE 0.25) and the full model (MSE 0.2475), the R2
| is (0.25-0.2475)/0.25=0.01. His calculation is slightly
| different: instead of 0.25-0.2475 he calculates directly
| 0.05^2 which is the variance of the predictions (in this
| case the total variance 0.25 can be decomposed as the
| variance of the errors 0.2475 plus the variance of the
| predictions 0.0025).
| fjkdlsjflkds wrote:
| (After re-reading the blog post with more care...) you
| are right, and thanks for the correction.
|
| Either way, the point stands... the improvement in using
| a full linear model (that predicts 0.45 or 0.55,
| depending on state) is marginal compared to the baseline
| model that always predicts 0.50, as you demonstrate with
| your code.
|
| To me, this doesn't seem paradoxical... the predictor is
| indeed providing little information over the "let's flip
| a coin to predict someone's voting preference"
| null/baseline predictor, since people's preferences (in
| aggregate) are almost equivalent to "flipping a coin".
|
| note: I meant "sum", but it's the same, since the ratio
| between sums of squares is equivalent to the ratio
| between mean squares
| kgwgk wrote:
| > Either way, the point stands... the improvement in
| using a full linear model (that predicts 0.45 or 0.55,
| depending on state) is marginal compared to the baseline
| model that always predicts 0.50
|
| Yes, I think we don't disagree. I was just puzzled by the
| "little variance left to explain" remark.
|
| > note: I meant "sum", but it's the same, since the ratio
| between sums of squares is equivalent to the ratio
| between mean squares
|
| You're right, sum of squares made sense if it was just
| for the ratio.
| fjkdlsjflkds wrote:
| Thanks for taking the time to clarify my confusion.
|
| It's not that there is "little variance left to explain",
| but actually that (no matter what) there will always be
| _too much_ variance left to be explained, when the
| response is Bernoulli-distributed and the parameter is
| not too far from 0.5 (i.e., the data generating process
| is like flipping a slightly loaded coin).
|
| If you use the expected value to predict the Bernoulli
| variable, you will always be somewhat wrong (0.45 and
| 0.55 are both far from 0 and from 1, which are the only
| possible responses).
|
| If you use a binary response to predict, you will quite
| often be _very_ wrong, even if you are right on average,
| and even if your prediction is to generate Bernoulli-
| distributed samples from the exact same distribution
| (i.e., you _know_ exactly how the coin is loaded /biased
| and you can exactly replicate its data generation
| process).
|
| So... yeah... no "paradox" ;)
| justk wrote:
| This is a little strange, you are using a data.frame with
| only two points so any linear model with two different
| parameters will be 100% accurate. This is the line that
| connect two points.
| fjkdlsjflkds wrote:
| An affine model, yes (as I mentioned in my comment). A
| linear model, no ;)
|
| But, anyway, seems like I interpreted things incorrectly.
| kgwgk wrote:
| Not sure about the "R squared" name, but the use of this
| quantity as "coefficient of determination" (the proportion
| of the variance in the dependent variable that is
| predictable from the independent variables) can be traced
| back to Sewall Wright's "Correlation and causation" (1921).
| jncfhnb wrote:
| When you're making a linear regression you're minimizing
| the sum of squared error which is 1:1 with R2. That is,
| you're getting the best possible R2 achievable with a line.
|
| The reason we do that is because we are assuming the errors
| are normally distributed and finding the slope that gets
| the best possible R2 is equivalent to getting figuring out
| how to fit the line with the maximum likelihood estimator
| of the error (aka mean of the distribution).
|
| So ultimately it's about curves. If you wanted to get a
| sense for why this is strongly desirable you should try to
| fit a linear regression using absolute error instead of
| squared error.
| jvanderbot wrote:
| Fun story: We once had a robot at a very big company
| you've all heard of that kept getting too close to walls
| when moving down hallway-like areas. The controls
| engineer swore he had tuned the controller to death and
| no further improvements could be had.
|
| A perception engineer took one look at the controller and
| saw linear error terms for distance left + distance right
| (distance to walls), changed it to distance left^2 +
| distance right^2, and the whole thing magically worked
| beautifully. Exercise for the reader: What position in
| the hallway minimizes sum of distances squared, vs what
| position(s) in the hallway minimize sum of distances
| without square.
|
| This is essentially the same problem you pose.
| jncfhnb wrote:
| Kind of funny but wouldn't you want to just
| maximize(distance_left, distance_right) if you wanted the
| center?
|
| Edit: no, derp, just walked into the same problem lol.
| Maximize the min should work though
| jvanderbot wrote:
| That has a unique solution, so it is infinitely better
| than the linear error, but is not a nice differentiable
| signal suitable for controls, I'd bet. But that wasn't in
| the problem statement.
| CamperBob2 wrote:
| Seems like you'd want to minimize the distance to the
| midpoint, but I'm probably missing something.
| jvanderbot wrote:
| Fun fact: that's what sum of square does
| dxbydt wrote:
| using just the ABS gives you the LAD. least absolute
| deviation is not used for a couple of reasons. LAD doesn't
| have a unique solution - you can code it up and your
| program will give you a set of solutions which will all
| minimize the LAD. this looks like a feature, but for
| various pedagogical reasons, statistians insist on BLUE.
| LAD can give you the LUE ie linear unbiased estimator, but
| not the B ie best. also, LAD doesn't have a closed form. so
| you can't write an equation and say this is the LAD, then
| differentiate it and derive interesting properties about
| the estimator. since there's no closed form, you can use
| numerical methods to find something that's good enough like
| error below 1e-6 and declare that to be the LAD. but that
| will happen at a bunch of points so no uniqueness.
| QuesnayJr wrote:
| The idea of using the square is due to Carl Friedrich Gauss
| when calculating the orbit of Ceres. I suspect he was
| familiar with the idea of absolute value.
| mihaaly wrote:
| > predicting individual votes
|
| Isn't that around the top of the statistics no-nos list?
| Probability theory in general? Predicting individual result
| based on the whole sample base? It was long ago and I am not in
| this field at all but my recollection tells me that it was
| mentioned in the beginning of the first class of probability
| theory 101.
| lupire wrote:
| No, it's the opposite. The sample space measurement gives you
| the probability of an individual selected item. This is the
| fundamental reason why probabily works.
|
| But it requires uniform sampling.
| kylebenzle wrote:
| Yes, you are right and statistics is confusing from the
| outside.
|
| My opinion is LLMs are just applied statistics and you see
| people losing their minds thinking the models have "come to
| life". Most people just really have no intuition for stats.
| c76 wrote:
| Isn't the phenomenon just related to the way the vote options are
| encoded? Use different methods and you will see different R^2
| results. Aren't the votes represented artificially on a
| continuous domain for the R^2 calculation but the actual values
| are categorical values?
| ayhanfuat wrote:
| Not really. No matter how you encode extremes will be 0% and
| 100% and one option will be 45% and the other 55%.
| parpfish wrote:
| part of this has to do with the fact that our intuitive sense of
| effect sizes don't really use proportions and we subconsciously
| start including sample sizes.
|
| If a state had an election with millions of voters and got a
| 55-45 result, it would be a decisive landslide victory; If a
| elementary school classroom had an election 20 voters had got a
| 55-45 split, it's be the narrowest possible margin of victory.
|
| Most would likely say that the effect in the former 'feels' much
| larger even though proportions are identical, which suggests that
| under the hood we're factoring in sample size to our intuition
| about effect sizes (probably something chi-square-ish).
|
| The result is that the framing of the problem can change our
| sense of how big the effects are. When we hear that these are
| state-level elections, we think it's a huge effect and feel that
| we should be able to do reverse inference. If it was reframed as
| an election on a much smaller sample, the paradox disappears and
| you'd say "of course you wouldn't be able to reverse that
| inference"
| thaumasiotes wrote:
| This ties in to the difference between whether an effect is
| statistically significant ("does the effect exist?") and
| whether it's significant ("does the effect matter?").
|
| It's very common to confuse the two ideas.
|
| In particular, in an election with many millions of votes and a
| 55-45 margin, it's common to describe the winner as receiving a
| mandate to rule, because it was so easy to determine who the
| winner was, despite the fact that they appear to be extremely
| unpopular. That's not a mandate in any ordinary sense.
| fjkdlsjflkds wrote:
| > whether an effect is statistically significant ("does the
| effect exist?") and whether it's significant ("does the
| effect matter?")
|
| The more accurate terms to describe this is whether an effect
| is _significant_ (i.e., "do we have enough information to be
| able to claim it is different from zero?") vs. whether an
| effect is _relevant_ (i.e., "is [our estimate of] the size
| of the effect meaningfully different from zero?").
|
| Statistics can only address the first issue (effect
| significance); the second issue (effect relevance) requires
| domain knowledge beyond statistics (in some contexts, a
| difference of 0.01 units can be irrelevant, while it may be
| relevant in other contexts).
| blt wrote:
| I'm no statistician, but the whole premise seems mismatched. Why
| are we using a tool from regression to analyze a classification
| problem?
| greesil wrote:
| Yep. I'm also not a statistician, but linear regression that
| the blogger is using predicts the mean for each state, and this
| is being conflated with trying to predict p(color | state). The
| goodness of fit here would be better modeled cross entropy and
| not a standard deviation.
| kgwgk wrote:
| For what it's worth, the "blogger" is an statistician.
| ayhanfuat wrote:
| I mean, Andrew Gelman, he is quite the statistician.
| greesil wrote:
| Oh he only went to MIT, pfff. And wrote a textbook.
| Xcelerate wrote:
| Haha, this is why I always Google whoever the author is
| to articles posted on HN before commenting. More than
| once I've thought "this person is an idiot" only to
| Google their name and find out they are a famous person
| in that field. Then I go back and re-read their article
| and realize I missed something more subtle going on.
| lupire wrote:
| This seems to be a special case.
|
| The blog post was rebutted by Seth in the comments 2
| weeks ago, same as Colin Percival's HN rebuttal, and
| Andrew didn't reply. It seems like a weird goof. Andrew
| was "buggin".
| chipdart wrote:
| > Why are we using a tool from regression to analyze a
| classification problem?
|
| Because classification is a regression problem.
|
| Think about it for a second. You want to put together a tool to
| tell which class an input belongs to. You have training data
| you can use to build your tool around. Your training data is
| already divided into sets that belong to a specific classm Your
| goal is to put together a model that can tell you what's the
| closest class your input belongs to by comparing with how close
| your input is to elements of the training data belonging to a
| specific class.
|
| What's your strategy?
|
| Well, one of the textbook strategie starts by specifying how
| you measure the distance between elements of your training set,
| and from that point you work on putting together a function
| that not only minimizes the distance between elements of your
| training set but also, when used to evaluate elements of a
| training set, works well in telling the type of elements of the
| training set that are closest to them. Then you assume the
| class of your input element is the same as the class of the
| elements of the training set that are closest to them.
|
| In the example above, the minimization step is... Yes,
| regression. You use regression to fit your model to your
| training data so that it is able how close your input element
| is to elements of a certain class, and then outputs how close
| it is to each of the classes.
| jncfhnb wrote:
| Classification models break down to regression problems under
| the hood, but regression metrics are not good tools to
| evaluate the efficacy of classification models.
| chipdart wrote:
| > Classification models break down to regression problems
| under the hood, but regression metrics are not good tools
| to evaluate the efficacy of classification models.
|
| You're simply wrong. Regression is a tried and true
| classification technique. Posting personal and baseless
| assertions don't refute that. I mean, pick up pretty much
| any textbook on supervised and unsupervised learning. You
| always end up with an approach which boils down to having
| training data, put together a trial function, apply a
| minimizer to fit trial functions to training data, and
| evaluate the resulting model by running trial data through
| it. Fitting trial functions to training data has a name:
| regression. Minimum squares has a very precise
| interpretation both in linear models and in probability.
| There is no way around it.
| jncfhnb wrote:
| In most modern ML circles "regression" analysis refers to
| the prediction of continuous variables whereas
| "classification" refers to the estimation across discrete
| outputs. This is true even for "logistic regression".
|
| What I said was that regression metrics are not good for
| evaluating usefulness in classification problems. An R2
| of 0.01. The fact that there are mitigating circumstances
| to justify why this might not be the case is not evidence
| that R2 is still a good thing to use. It's actually
| evidence of the opposite.
|
| With classification problems we are concerned more often
| with the ordinality of estimated probabilities vs
| outcomes and/or the calibration of a model.
|
| > Posting personal and baseless assertions don't refute
| that.
|
| Your mom didn't refute me either.
| chipdart wrote:
| > In most modern ML circles "regression" analysis refers
| to the prediction of continuous variables (...)
|
| You're voicing very opinionated takes while showing
| considerable ignorance on the topic.
|
| Classification problems are solved with trial functions
| adjusted to the training set through regression. These
| trial functions,once fitted, represent membership
| functions. They are essentially interpolation functions
| that, say, converge to 1 when close to elements of the
| training set of a specific class, and 0 for members of
| all other classes. In grey areas where elements of the
| training set are sparsely distributed, these membership
| functions can output values in the middle, because they
| are interpolating.
|
| That's literally data mining 101.
|
| > What I said was that regression metrics are not good
| for evaluating usefulness in classification problems.
|
| That's simply not true. It's like saying models that do
| not fit the data are good approximations of the data.
| Another way to put it is praising the accuracy of a
| broken clock because it's spot on two times a day.
|
| > The fact that there are mitigating circumstances to
| justify why this might not be the case is not evidence
| that R2 is still a good thing to use. It's actually
| evidence of the opposite.
|
| I don't think you have an adequate grasp on the subject,
| neither classification problems nor linear models.
| Therefore, your personal baseless assertions don't mean
| anything nor bring any value to the discussion.
| vcdimension wrote:
| I am a statistician, and you're right, for this kind of thing
| we would normally use a binary response model such as a logit
| or probit model that constrains the response variable to be
| between 0 & 1. However in this case it doesn't matter since
| there's only one independent variable (state), and it's binary
| so there's only 2 different predictions the model could make
| (which will be the correct probabilities of 0.45 & 0.55, even
| with a linear model).
|
| The normal R^2 formula can't be applied to a logit/probit
| model; instead you use an alternative such as McFadden's or Cox
| & Snell pseudo R-squared. I'd be interested to see what value
| they take for this example.
|
| Linear models are sometimes used even in models with many
| independent variables since it can be shown that the
| coefficients in a linear model are unbiased estimators for the
| average partial effects of any non-linear binary response
| model.
| leto_ii wrote:
| As other commenters have pointed out in one way or another, the
| problem seems to actually be that this simplistic model of voter
| choice can't capture all the structure of the real world that
| humans can quickly infer from the setup. Things like: state
| elections have millions of voters, 55/45 is actually a decisive,
| not a narrow win etc.
|
| In a generic setup, imagine you have a binary classifier that
| outputs probabilities in the .45-.55 range - likely it won't be a
| really strong classifier. You would ideally like polarized
| predictions, not values around .5.
|
| Come to think of it, could this be an issue of non-ergodicity too
| ( hope I'm using the term right)? i.e. state level prior is not
| that informative wrt individual vote?
| jncfhnb wrote:
| No, you want your model to be well calibrated. If the model
| accurately assessed a 0.55 probability of going blue, then that
| is what you want.
|
| People who try to correct for "unbalanced classes" and contort
| their model to give polarizing predictions are frankly being
| pretty dumb.
|
| The correct answer is to take your well calibrated
| probabilities and use you brain on what to do with them.
| leto_ii wrote:
| This is not a matter of class balance that much. If you want
| to predict which of two parties somebody will vote with, the
| most natural framing is that of binary classification.
|
| For that you need to threshold your predictions. Ideally
| you'd like your model to generate a bimodal distribution so
| that you can threshold without many false positives etc.
| jncfhnb wrote:
| Yes but the prompt here states that all we know is the
| probability is either 0.55 or 0.45. By definition this is
| the best model you can produce.
| gpsx wrote:
| My statisitcs are a little rusty, so I might be off here. Someone
| correct me if I have this wrong. R^2 = 1 would be every voter in
| one state votes blue and every voter in the other votes red. R^2
| = 0 would mean both states are exactly even between red and blue.
| The states are a lot closer to that. Again, my statistics are
| rusty so I'm no sure if this next part is valid, but sqaure root
| of .01 is .1, which doesn't seem like such a bad representation
| of the situation.
| kazinator wrote:
| Nothing but endless cloudfare captchas here for me.
|
| Removing cookies for the domain doesn't help, because (doh) I've
| never visited it before.
| dash2 wrote:
| This comes up a lot in genetics. One crowd says "polygenic scores
| for education don't tell you much, because look how low the
| R-squared is!" Another crowd (including me) says "polygenic
| scores for education are a big deal, because look how big the
| effect size is!"
| justk wrote:
| The math is correct, but I think the model used is not correct
| since it doesn't reflect that the variable s is dichotomous so
| rather a mixed model should be used. If we continue thinking that
| s is continuous we could think of this example: s=state is
| encoded as a continuous variable between -1 and 1 here people
| change state frequently and -1 reflects the person will vote in
| the blue state with probability 1 and s=1 that the person will
| vote in the red state with probability 1 while s=0 means that the
| person has the same probability of voting in the red or blue
| states. When s is near zero the model is not able to predict the
| preferences of the voter and this is the reason of the low
| predictive power of this model for a continuous s. The extreme
| cases s=-1 or s=1 could be rare for populations that move from
| one state to the other frequently so the initial intuition is
| misleaded to this paradox.
| mtts wrote:
| This.
|
| R2 is not the correct measure to use.
|
| This article is a perfect example of the principle that simply
| doing math and getting results is not necessarily meaningful.
| kgwgk wrote:
| R2 is a measure like any other. In this case it measures the
| relative reduction in MSE - which is low because the
| prediction of individual votes remains quite bad even if the
| state is taken into account.
|
| Does another measure give substantially different results?
| justk wrote:
| I think that you are using here a different definition of
| R^2 for example the way you are thinking of R^2 doesn't
| allow for an interpretation of the constant term used in
| the linear model for the formula of the R^2 to be true.
| What you are thinking is R^2 = 1 - mean(the variance in
| each state)/(total variance), but that is not the
| definition of R^2 for a linear model.
|
| As the user fskfsk.... says in another comment, here the
| constant term explains a lot of the variance so that the
| slope terms contains less information, that is not
| available using your definition or idea of R^2
| kgwgk wrote:
| > I think that you are using here a different definition
| of R^2
|
| Different from what?
|
| According to wikipedia:
|
| The most general definition of the coefficient of
| determination is R^2 = 1 - SS_res / SS_tot ( = 1 - 0.2475
| / 0.25 = 0.01 in this case)
|
| Edit to clarify the definition above:
|
| SS_res is the sum of squares of residuals (also called
| the residual sum of squares) [?]( y_i - predicted_i )^2
|
| SS_tot is the total sum of squares (proportional to the
| variance of the data) [?]( y_i - [?]y_i/N )^2
| justk wrote:
| The most general definition of R^2 can produce a result
| that is negative, and we are talking about a paradox
| related to values of R^2 that one should expect. So it is
| common to use linear models and linear regression. I
| don't know if the variance of the total population can be
| computed as the sum of the variances in each state, and
| state is not a continuous variable.
|
| The population variance is the sum of the Between Group
| Variance and the Within Group Variance weighted by the
| number of elements in each group.
| kgwgk wrote:
| I don't understand what you mean. I'll just note that the
| value of R2 in this case is 1% as the blog post explains
| and the code below confirms. > data <-
| data.frame(state = rep(c(0, 1), each=20), pref = c(rep(0,
| 11), rep(1, 9), rep(0, 9), rep(1, 11))) >
| summary(lm(pref ~ state, data = data))$r.squared
| 0.01
| justk wrote:
| The math is correct, I am referring to your comment: >>
| R2 is a measure like any other. In this case it measures
| the relative reduction in MSE - which is low because the
| prediction of individual votes remains quite bad even if
| the state is taken into account.
|
| I may be reading too much from your comment, but it seems
| that you relate R^2 to the reduction in the prediction
| error in each state, so it seems you are thinking about
| the formula of computing the R^2 as the (average variance
| in each state)/(total variance), that I think is not
| correct in general since at least it should require the
| total variance to be the sum of the variances in each
| state. If you based your ideas in that formula then your
| intuition is not correct, that is my point. When I apply
| R^2 I am thinking in a multivariable linear model with
| continuous variables, and this is not the case. I should
| measure this problem by how the entropy change when we
| apply the information about the state, something like the
| cross entropy using the total distribution and the
| distribution by states.
| kgwgk wrote:
| When I wrote "In this case it measures the relative
| reduction in MSE" I meant exactly that.
|
| The mean squared error of the baseline model which
| doesn't include the state as a regressor is 0.25 (it
| predicts always 0.5 - it's off by 0.5 in every case).
|
| The mean squared error of the model which includes the
| state as a regressor is 0.2475 (it predicts 0.45 or 0.55
| depending on the state - in both cases it's off by 0.45
| with 55% probability and it's off by 0.55 with 45%
| probability).
|
| The mean squared error is directly related to variance
| when the predictor is unbiased. The ratio of the sum of
| squares is the same as the ratio of the mean square
| errors.
|
| Edit: http://brenocon.com/rsquared_is_mse_rescaled.pdf
|
| "R2 can be thought of as a rescaling of MSE, comparing it
| to the variance of the outcome response."
|
| https://dabruro.medium.com/you-mention-the-average-
| squared-e...
|
| "Also it is worth mentioning that R-squared (coeff. of
| determination) is a rescaled version of MSE such that
| 100% is perfection and 0% implies the same MSE that you
| would get by simply always predicting the overall mean of
| the dataset."
| justk wrote:
| Let d1 = data[state==0] and d2 = data[state==1], then
| var(d1$pref) = 0.26, var(d2$pref)= 0.26 and var(d$pref)=
| 0.256 (using R and one of your dataframes), so the
| intuition is that knowing the state does not give
| information about the preferences of the voters, so this
| suggests that any model based on state should give poor
| results and so having R^2=1 is not a big paradox in this
| case.
|
| There must be a formula to compute R^2 from variances
| both among states and inside states but anyway, when the
| variances inside any state are bigger that the total
| variance that should imply that the feature that divides
| the population in groups is of little value for
| prediction so it should have a small R^2 value.
| kgwgk wrote:
| That seems more or less what I said in the comment you
| replied to: the prediction of individual votes remains
| quite bad even if the state is taken into account. That's
| why the relative reduction in MSE is low. That's why the
| R2 is low. I don't think there is any paradox.
|
| I was replying to someone who claimed that "R2 is not the
| correct measure to use. This article is a perfect example
| of the principle that simply doing math and getting
| results is not necessarily meaningful." I've not seen any
| comment from anyone getting "different results" with a
| different measure.
|
| Edit: You used var(...) which includes a factor N/N-1 and
| doesn't give exactly the total sum of squares.
|
| The example dataframe contains 40 observations (20 per
| state) and you get higher variance estimate for the
| subsamples than for the aggregate sample but if you put
| toghether a few copies of the data (for example doing
| "data <- rbind(data, data, data, data, data)") even the
| adjusted (unbiased) estimator of the variance is lower
| for the states.
|
| You can calculate the "exact" values yourself doing
| (x-mean(x))^2 or undoing the adjustment:
| > var(data$pref)*39/40 [1] 0.25 >
| var(data[data$state==0, "pref"])*19/20 [1] 0.2475
| > var(data[data$state==1, "pref"])*19/20 [1] 0.2475
|
| > when the variances inside any state are bigger that the
| total variance
|
| They are not. But you're right in that a small difference
| shows that dividing the population in groups is of little
| value for prediction and that's why the R^2 value is
| small.
| justk wrote:
| The correcting factor n/(n-1) in R is what explains my
| paradox about the law of total variance Var(Y) =
| E(var(Y|X)) + Var(E(X|Y)), I was obtaining result that
| don't match this formula because I corrected all the
| variances with the factor 20/19 but the total variance
| should have the factor 40/39 just like you pointed.
| Thanks for the comments and the correction.
|
| I just added another comment that relates analysis of
| variance to this post to show that there is no real
| paradox here.
|
| Finally, the formula for the total variance above is
| related to my intuition that having some information
| (having the data for each state) should make the means of
| the variances in each group smaller that the total
| variance, because variance is related to lack of
| information. But analysis of variance suggests (see other
| comment of mine) that the state factor is not
| representative because the high variance in each group
| (each state) and the low difference between the groups
| means and the total mean.
| jncfhnb wrote:
| A mixed model is not relevant here. A simple linear regression
| with one variable will achieve exactly the same results. Coding
| it as -1 and 1 has no difference to coding it as 0 and 1. You
| just stuff the rest into the intercept.
|
| You would also want to be predicting 0.45 and 0.55 not 1 and 0
| because we solve for squared error.
| jncfhnb wrote:
| R2 is more simply explained as the share of the error variance
| explained by the model out of the share of the error explained by
| the best guess, which is, in this case 0.5.
|
| Guessing 0.5 will have you wrong wrong by 0.5 100% of the time.
| SST is 25 for a 100 sample example.
|
| Guessing 0.55 for the 0.55 state will have you wrong by 0.45 55%
| of the time and 0.55 45% of the time for the other. SSE is 24.75
|
| 1- 24.75 / 25 = 0.01
|
| Looking at it this way it's not too hard to see why the R2 is
| bad. It barely explains any more difference in the individual
| behavior than the basic guess.
|
| R2 is not a great metric for percentages or classification
| problems like this.
| fosdad2131321 wrote:
| There are two ways to resolve the paradox
|
| 1. if you insist on using the r-squared (i.e., a linear
| regression measure), then properly center and normalize your
| data, and model what you actually predict: the difference between
| the baseline (0.5) and the probability to vote for party 0 or
| party 1. If you model the outcomes as 0/1 without this, then you
| are using a model made for gaussian variables on what should be a
| logistic regression 2. if you can live with something that more
| accurately captures the idea of "explanatory power", you can use
| a GLM (logistic link function), do a logistic regression, and
| then use the log odds or another measure.
|
| In both cases, the variance explained by the state that you are
| in is 1, because of course it is, that's how the thought
| experiment is constructed - p(vote for party 1)=0.5+
| \delta(state).
|
| "Paradoxes" like this are often interesting in the sense that
| they point to the math being the wrong math or you using it
| wrong, but instead people tend to assume that they are obviously
| understanding things correctly so it must be some weird property
| of the world (which then sometimes is used to construct some
| faulty conclusions as in some of the cited papers)
| justk wrote:
| From (1) On the other hand, if the variation between the group
| means and the grand mean is small, and the variation within
| groups is large, this suggests there are no real differences in
| the group means i.e. the variations we observe is just sampling
| variation.
|
| The above is in the context of analysis of variance. In our
| example the means in each state are 0.55 and 0.45 and the total
| mean is 0.50 so first summand is small but the variances in the
| red and blue states are both 0.247, large summand, so the
| variations we observe are just sampling variations. Hence the
| state factor is not important and that explains the low R^2
| value. Note that in each state the predicted value for the model
| is the group mean of that group. So analysis of variance explains
| that the OP result is not a paradox or something strange.
|
| https://saestatsteaching.tech/analysis-of-variance
| gweinberg wrote:
| The states (and even more so the sub-state regions) really are
| much more different than what you would think just looking at R
| vs D. A Democrat in a city the Democrats win 90-10 is likely a
| very different Democrat from one where they lose 60-40.
| Chinjut wrote:
| If you think that's bad, the R^4 coefficient is even lower.
___________________________________________________________________
(page generated 2024-07-04 23:01 UTC)