https://stats.stackexchange.com/questions/674129/why-does-a-linear-least-squares-fit-appear-to-have-a-bias-when-applied-to-simple Skip to main content Stack Exchange Network Stack Exchange network consists of 183 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers. Visit Stack Exchange [ ] Loading... 1. + Tour Start here for a quick overview of the site + Help Center Detailed answers to any questions you might have + Meta Discuss the workings and policies of this site + About Us Learn more about Stack Overflow the company, and our products 2. 3. current community + Cross Validated help chat + Cross Validated Meta your communities Sign up or log in to customize your list. more stack exchange communities company blog 4. 5. Log in 6. Sign up Cross Validated 1. 1. Home 2. Questions 3. Unanswered 4. AI Assist 5. Tags 6. 7. Chat 8. Users 9. 2. Stack Internal Stack Overflow for Teams is now called Stack Internal. Bring the best of human thought and AI automation together at your work. Try for free Learn more 3. Stack Internal 4. Bring the best of human thought and AI automation together at your work. Learn more Stack Internal Knowledge at work Bring the best of human thought and AI automation together at your work. Explore Stack Internal Why does a linear least squares fit appear to have a bias when applied to simple test data? Ask Question Asked 2 days ago Modified 2 days ago Viewed 5k times 14 $\begingroup$ I used python to generate a correlated data set for testing, and then plotted a basic linear least-squares fit. The result looked a bit strange to me, because the line doesn't really seem to pass "centrally" through the data. It looks a little "tilted": enter image description here So, instead, I then diagonalized the covariance matrix to obtain the eigenvector that gives the direction of maximum variance. This is shown by the black arrow in the figure below. This one does indeed point in the direction that I would expect: enter image description here So, I am looking for a bit of an intuitive explanation for what is going on here. I know that they are not measuring the same thing - the fit is minimizing the sum of the squared "errors", the vertical distances between the measured values and the fitted model. Whereas the eigenvector is chosen to maximize variance. But I guess I am surprised by the result - I would've expected the fit to go through the center of the data cluster, and not have this bias. Is it something to do with the fact that minimizing the vertical distances somehow breaks the symmetry in the system? Is it inappropriate to apply a basic linear fit to such data? Here is the code: import numpy as np import matplotlib.pyplot as plt def get_correlated_dataset(n, dependency, mu, scale): latent = np.random.randn(n, 2) dependent = latent.dot(dependency) scaled = dependent * scale scaled_with_offset = scaled + mu return scaled_with_offset[:, 0], scaled_with_offset[:, 1] # Generate Data dependency = [[30, 30], [30, 2]] mu = [150,-100] scale = 1 x, y = get_correlated_dataset(20000, dependency, mu, scale) # Calculate direction of maximum variance (eigenvector of covariance matrix) covariance_matrix = np.cov( np.stack((x, y), axis=0) ) EIGVALS,EIGVECS = np.linalg.eig( covariance_matrix ) vec = EIGVECS[:, np.argmax(EIGVALS)] # Get the eigenvector corresponding to largest eigenvalue # Magnitude of vector, 3 standard deviations, and mean (for plotting arrow) vec_norm = np.sqrt(vec[0]**2 + vec[1]**2) sigma3 = np.sqrt(EIGVALS.max())*3 mu_x, mu_y = np.mean(x), np.mean(y) # Linear fit to the data slope, intercept = np.polyfit(x, y, 1) fig1,ax1 = plt.subplots(figsize=(8, 5)) ax1.plot( x , y ,'o',label="Generated Data") ax1.plot( x , x*slope+intercept ,label="Linear Fit") ax1.quiver(mu_x, mu_y, sigma3*vec[0]/vec_norm, sigma3*vec[1]/vec_norm, zorder=10,angles='xy', scale_units='xy',scale=1,label="Eigenvector of Covariance Matrix") * regression * pca * covariance * fitting Share Cite Improve this question Follow edited 2 days ago User1865345's user avatar User1865345 12k1313 gold badges2727 silver badges4343 bronze badges asked 2 days ago teeeeee's user avatar teeeeee 31933 silver badges1212 bronze badges $\endgroup$ 15 * 1 $\begingroup$ I don't have any idea what is going on here, but there is a lot of overplotting in your graphs. You might want to plot either smaller dots or unfilled circles. I don't think this is the real issue here, but it seems like good practice $\ endgroup$ Peter Flom - Peter Flom 2026-01-02 12:21:00 +00:00 Commented 2 days ago * 5 $\begingroup$ This seems like the difference between total-least-squares and ols (i.e. ordinary least squares). We have a few relevant threads on the topic, including this recent one: stats.stackexchange.com/questions/672804/... $\endgroup$ mkt - mkt 2026-01-02 12:27:59 +00:00 Commented 2 days ago * 4 $\begingroup$ This paper is also very relevant here: humans tend to overestimate the slope of a regression line when making a "visual fit"; we intuit more towards orthogonal least squares than ordinary (as mkt also mentioned). Importantly, the latter assumes no error exists along the X-axis. Eigen decomposition is effectively total least squares, so this confirms the bias described in the paper. $\endgroup$ PBulls - PBulls 2026-01-02 12:44:00 +00:00 Commented 2 days ago * 8 $\begingroup$ Its important to know that the ordinary regression line doesnt try to fit the major axis of the bivariate ellipse-shape; instead it estimates the conditional expectation of Y|X=x, which is somewhat nearer to the horizontal. See the second plot here for an illustration: stats.stackexchange.com/a/ 226054/805 (indeed, the question and answers there may help you) $\endgroup$ Glen_b - Glen_b 2026-01-02 13:32:54 +00:00 Commented 2 days ago * 3 $\begingroup$ @Ben It duplicates a lot of material (I linked to a search in my earlier comment), but I have been reluctant to close the thread due to the interest it has been generating. $\ endgroup$ whuber - whuber 2026-01-02 19:28:30 +00:00 Commented 2 days ago | Show 10 more comments 5 Answers 5 Sorted by: Reset to default [Highest score (default) ] 19 $\begingroup$ The regression line does, in fact, go through the "center", but since the erorrs are measured vertically, so is the "center". I think this image shows it quite clearly (see the black vertical chord): enter image description here Share Cite Improve this answer Follow answered 2 days ago Steve's user avatar Steve 33611 silver badge33 bronze badges $\endgroup$ 1 * 4 $\begingroup$ This is the best visualization in this question so far. $\endgroup$ justhalf - justhalf 2026-01-03 03:47:01 +00:00 Commented yesterday Add a comment | 15 $\begingroup$ There are already loads of great answers here but I feel like it's necessary to add that TLS/the main PCA/ eigenvector, however you want to call it, can be something very different from what we normally understand as a linear regression line. They just convergence towards each other the closer you are to the conditions in @Dominic Scoccheras Answer. Consider: $ X \sim \mathcal N(0,1) \text{ and } Y = 0.5X+e, e\sim\mathcal N (0,2) $ The resulting picture with n = 20000 looks like this:scatter plot with low alpha and an extreme resolution to properly represent the high variance of y. Additional red elippsis and main-pca vs regression line in black. If you wanted to use the main PCA, i.e. the red arrow, to predict $E [Y| X=2]$ you'd end up way of the scale at 26, while correct value is of course 1! Here's the R-code: library(ellipse) n <- 20000 x <- rnorm(n) res_sd <- 2 y <- 0.5*x + rnorm(n, sd = res_sd) eigen(M)[["vectors"]][, 1] draw_eigen_arrow <- function(M, color){ eigen_vec <- eigen(M)[["vectors"]][, 1]*res_sd arrows(0, 0, eigen_vec[1], eigen_vec[2], col = color, lwd = 2) } png(width = 500*sd(x), height = 500*sd(y)) plot(x, y, col = rgb(0,0,0, alpha = 0.25)) M <- matrix(c(1, 0.25, 0.25, 0.25 + res_sd^2), nrow = 2) abline(a=0, b=0.5, lwd = 2) points(ellipse(M), col = 2, lwd = 2, type = "l") draw_eigen_arrow(M, 2) dev.off() eigen_vec <- eigen(M)[["vectors"]][, 1] eigen_vec[2]/eigen_vec[1]*2 Share Cite Improve this answer Follow answered 2 days ago Lukas Lohse's user avatar Lukas Lohse 5,16366 silver badges2828 bronze badges $\endgroup$ Add a comment | 12 $\begingroup$ So, I am looking for a bit of an intuitive explanation for what is going on here. The "intuitive" TL;DR is that OLS as a model inherently treats the dependent and the independent variable asymmetrically: only $y$ is assumed to be noisy, while $x$ is assumed to be exact. I would've expected the fit to go through the center of the data cluster. I'm not sure where that expectation is violated. The OLS fit does go through the "center", as long as you define the center as the point $ (\mu_x, \mu_y)$. Share Cite Improve this answer Follow answered 2 days ago arpad's user avatar arpad 2,09355 silver badges1515 bronze badges $\endgroup$ 3 * $\begingroup$ When I say "go through the center", I don't just mean passes through a center point at one place. What I really mean is "line is placed through the region of largest density", or "passes through the most number of points". This is what I think of as being a "best-fit" line. $\endgroup$ teeeeee - teeeeee 2026-01-02 13:17:22 +00:00 Commented 2 days ago * 4 $\begingroup$ That's no doubt your expectation, but the least-squares fit pays no attention to local density and it is not guaranteed to go through any data points. The criterion is just in terms of a sum of squared errors to be minimised. $\ endgroup$ Nick Cox - Nick Cox 2026-01-02 13:25:22 +00:00 Commented 2 days ago * 1 $\begingroup$ @teeeeee Ah, I see. Yes, that can indeed be another definition of center, although I'd say that the more common interpretation is a centroid point. $\endgroup$ arpad - arpad 2026-01-02 13:27:28 +00:00 Commented 2 days ago Add a comment | 11 $\begingroup$ As @mkt said this is the difference between OLS and total least squares. The solution to the slope of the line in OLS is $\beta_{ols} = {\sigma_{xy}} / {\sigma_{x}^2}$. What you also did was find the eigenvector with the maximum corresponding eigenvalue (PCA). For TLS we find the line with the smallest eigenvalue, which is the same line as the eigenvectors are orthogonal. The direction line from PCA is $\ mathcal{l}(t)=\mu+t v$ ($\mu=(\mu_{x}, \mu_{y})^{T}$, $v$ is the eigenvector $v=(a,b)^{T}$) and the TLS line is $\mathcal{l}(t)=\mu+t (-b,a)^{T}$. These lines differ from OLS as the slope of the PCA line is $\beta_{PCA}=\bigg[\sigma_{y}^{2}-\sigma_{x}^{2}+\sqrt{(\sigma_{y} ^{2}-\sigma_{x}^{2})^{2}+4\sigma_{xy}^{2}}\bigg]/{2\sigma_{xy}}$, which is only the same as the OLS slope if $(\sigma_{x}^{2}=\sigma_ {y}^{2})$ or $\sigma_{xy}=\pm\sigma_{x}\sigma_{y}$ (perfect linear relationship $|\rho|=1$). Here is your code edited to show this: import numpy as np import matplotlib.pyplot as plt def get_correlated_dataset(n, dependency, mu, scale): latent = np.random.randn(n, 2) dependent = latent.dot(dependency) scaled = dependent * scale scaled_with_offset = scaled + mu return scaled_with_offset[:, 0], scaled_with_offset[:, 1] # ---------------------------- # Generate data # ---------------------------- np.random.seed(0) dependency = [[30, 30], [30, 2]] mu = [150, -100] scale = 1 x, y = get_correlated_dataset(20000, dependency, mu, scale) mu_x, mu_y = np.mean(x), np.mean(y) # OLS ols_slope, ols_intercept = np.polyfit(x, y, 1) # PCA cov = np.cov(x, y) eigvals, eigvecs = np.linalg.eig(cov) v = eigvecs[:, np.argmax(eigvals)] pca_slope = v[1] / v[0] # TLS X = np.column_stack((x - mu_x, y - mu_y)) _, _, Vt = np.linalg.svd(X, full_matrices=False) direction = Vt[0] tls_slope = direction[1] / direction[0] print("OLS:", ols_slope) print("PCA:", pca_slope) print("TLS:", tls_slope) Share Cite Improve this answer Follow edited 2 days ago User1865345's user avatar User1865345 12k1313 gold badges2727 silver badges4343 bronze badges answered 2 days ago Dominic Scocchera's user avatar Dominic Scocchera 11144 bronze badges New contributor Dominic Scocchera is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct. $\endgroup$ 2 * $\begingroup$ Thanks, I'm not sure what it is about my generated test data which requires a method that allows errors in the x variable as well (i.e. total least squares)? $\endgroup$ teeeeee - teeeeee 2026-01-02 13:27:59 +00:00 Commented 2 days ago * 2 $\begingroup$ +1. Welcome to Cross Validated! @teeeeee there's nothing about your data that requires a "method that allows errors in the x variable." But if you choose to use ordinary least squares instead of total least squares (TLS) you will get the "surprising" result that you found. Your intuition agrees with the TLS result, which treats variance inx and y symmetrically. $\endgroup$ EdM - EdM 2026-01-02 16:38:59 +00:00 Commented 2 days ago Add a comment | 9 $\begingroup$ If you want an intuitive understanding of what you observed, you first need to realize, as mentioned already by several contributors, that OLS treats Y and X asymmetrically. To convince you of this, just plot, on your existing graph, in addition to the OLS line of Y against X, the OLS line of X against Y. You should see a different regression line. The two lines do in fact intersect at the point $(\bar x, \bar y)$ (that is a property of OLS; but that point may not be the center of gravity...); and you will find that your "intuitive" best fit is somewhere between these 2 lines. This is because OLS regression of Y against X minimizes the sum of vertical errors, while OLS regression of X against Y minimizes the sum of horizontal errors, but our visual intuition (our pattern matching brain) goes for minimizing the Euclidean (orthogonal) distance: hence the cognitive disconnect. The plot below illustrates the scenario (it is based on some data I had, looking at the 6-minute walk distance (6MWD) as a function of age), when I was making that very point. XYRegression You can clearly see that both OLS regressions are slightly "biased" wrt what one might call the "main axis". If you want a regression which matches your intuition, then look (as others have mentioned) at total least squares regression, or at Deming regression, or at orthogonal regression. So the short answer to the question posed in your title is, "because it is supposed to have a bias". Share Cite Improve this answer Follow edited 2 days ago Ben Bolker's user avatar Ben Bolker 55.8k44 gold badges154154 silver badges219219 bronze badges answered 2 days ago jginestet's user avatar jginestet 14.1k22 gold badges99 silver badges3737 bronze badges $\endgroup$ 1 * $\begingroup$ Great answer. When does it make sense, from a perspective of prediction, to "want" OLS vs TLS? $\endgroup$ Jonah - Jonah 2026-01-04 22:54:46 +00:00 Commented 5 mins ago Add a comment | Your Answer [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] Thanks for contributing an answer to Cross Validated! * Please be sure to answer the question. Provide details and share your research! But avoid ... * Asking for help, clarification, or responding to other answers. * Making statements based on opinion; back them up with references or personal experience. Use MathJax to format equations. MathJax reference. To learn more, see our tips on writing great answers. Draft saved Draft discarded [ ] Sign up or log in Sign up using Google Sign up using Email and Password Submit Post as a guest Name [ ] Email Required, but never shown [ ] Post as a guest Name [ ] Email Required, but never shown [ ] Post Your Answer Discard By clicking "Post Your Answer", you agree to our terms of service and acknowledge you have read our privacy policy. Start asking to get answers Find the answer to your question by asking. Ask question Explore related questions * regression * pca * covariance * fitting See similar questions with these tags. * Featured on Meta * Native Ads coming soon to Stack Overflow and Stack Exchange * A proposal for bringing back Community Promotion & Open Source Ads Linked 67 What is the intuition behind conditional Gaussian distributions? 25 Why isn't total least squares more widely used? 9 What does this plot tell me about my linear model? Related 4 Which statistics textbooks would explain why linear least squares regression fails when data is heteroskedastic 5 How to calculate bias when we have an estimation using simple linear regression? 5 Why does the total least squares line in 2D pass through the average across all data points? 0 Why does least squares refuse to overfit my data? 6 Why does ordinary least squares have to be linear in the parameters? 4 What does it mean to "fit" something using least squares? 3 Finding the right model for non-linear least squares fitting of data Hot Network Questions * The road trip of the century * What do dark and transparent part of character and machine stats mean? * Is there a way to leave the Azure Snow Land Warp to the left in 0%? * how do we actually reason/think/comprehend about nothing? * Did the widely-reported espionage story involving kefir really happen? * The Stack'O'Mounts? * Change Text Color of Certain Lines in Algorithmic Environment * Takagi: "In the case of a general function, such a procedure cannot be carried out. This is where the subtlety (or complexity) of derivatives arises." * How do copyleft licenses handle support routines in compilers? * In a brain with suspended consciousness, when does an unperceived event occur from the subject's perspective * Which nitrogen atoms of biguanide donate electron pair(s) to the metal ion? * (Probably) 1960/70s British SF novel about an invisible alien spaceship use dreams to cause humanity to destroy themselves * taxes on the sale of gifted gold coins * chto skazat' (as an idiom or introductory phrase) * Completeness in the box topology * Exported PS1 environment variable, doesn't persist in a subshell * Significance of the namajapa of Siva * Where I can read full grub2 doc * Can Wang (king) ever be a semantic component in kanji? * How are we to rightly understand Faramir's quote to Sam as relates to free will vs. determinism? * Why is it a "hybrid" rose in Rose's poem "The Subtle Briar"? * Supervisor / boss obsessed with ResearchGate and other academic social media platforms * Halving a 6x6 Grid into Two Identical Shapes * I need help with AnyDice.com to determine probabilities of a new system using d10 + Tarot Cards more hot questions Question feed Subscribe to RSS Question feed To subscribe to this RSS feed, copy and paste this URL into your RSS reader. [https://stats.stacke] * Cross Validated * Tour * Help * Chat * Contact * Feedback Company * Stack Overflow * Stack Internal * Stack Data Licensing * Stack Ads * About * Press * Legal * Privacy Policy * Terms of Service * Your Privacy Choices * Cookie Policy Stack Exchange Network * Technology * Culture & recreation * Life & arts * Science * Professional * Business * API * Data * Blog * Facebook * Twitter * LinkedIn * Instagram Site design / logo (c) 2026 Stack Exchange Inc; user contributions licensed under CC BY-SA . rev 2025.12.22.38265