6
$\begingroup$

Let's say I have a linear regression model of the form $ y = B_x x + I_x + \epsilon $, where $B_x$ is the beta coefficient of the $x$ term, $I_x$ is the intercept term and $\epsilon$ is additive, normally distributed noise. If I have a dataset and perform linear regression, I get a value for $B_x$, which indicates the slope of the relationship.

If I swap the roles of the $x$ and $y$ data, and try to fit a model of $x = B_y y + I_y + \epsilon$, I would expect intuitively that $B_y = \frac{1}{B_x}$. A simple geometric argument can be made to show that swapping the roles of $x$ and $y$ shouldn't change the position of the regression line w.r.t. any data point, and from here it seems like simple algebra that if $y = Bx + I$ then $x = \frac{1}{B} y + \frac{I}{B}$.

Where is this reasoning wrong? Can someone explain to me why $B_x \neq \frac{1}{B_y}$, preferably without resorting to tons of linear algebra or direct derivation from the normal equation?

Improve this question
$\endgroup$

4 Answers 4

Reset to default
3
$\begingroup$

Well, I think Mike McCoy's answer is "the right answer," but here's another way of thinking about it: the linear regression is looking for an approximation (up to the error $\epsilon$) for $y$ as a function of $x$. That is, we're given a non-noisy $x$ value, and from it we're computing a $y$ value, possibly with some noise. This situation is not symmetric in the variables -- in particular, flipping $x$ and $y$ means that the error is now in the independent variable, while our dependent variable is measured exactly.

One could, of course, find the equation of the line that minimizes the sum of the squares of the (perpendicular) distances from the data points. My guess is that the reason that this isn't done is related to my first paragraph and "physical" interpretations in which one of the variables is treated as dependent on the other.

Incidentally, it's not hard to think up silly examples for which $B_x$ and $B_y$ don't satisfy anything remotely like $B_x \cdot B_y = 1$. The first one that pops to mind is to consider the least-squares line for the points {(0, 1), (1, 0), (-1, 0), (0, -1)}. (Or fudge the positions of those points slightly to make it a shade less artificial.)

Another possible reason that the perpendicular distances method is nonstandard is that it doesn't guarantee a unique solution -- see for example the silly example in the preceding paragraph.

(N.B.: I don't actually know anything about statistics.)

Improve this answer
$\endgroup$
2
  • $\begingroup$ Right. I was aware in the back of my mind that vertical distance is what's used in regression and that the reason for this is that you the interpretation of a regression is that it gives you the best possible predictor of Y given X (minimum vertical distance) assuming the relevant assumptions are met. However, for some reason I kept visualizing the problem as using perpendicular distance. $\endgroup$
    – dsimcha
    Mar 24, 2010 at 14:05
  • $\begingroup$ imho, Paul Teetor gives a superb explanation of what is going on in, for example quanttrader.info/public/betterHedgeRatios.pdf . He is interested in 'spread trades' but the explanation he gives is universal and excellent. As others have said, the point is that ols minimizes 'x distance' which can be very different from 'y distance' particularly if the slope of the regression line is very close to zero. $\endgroup$
    – meh
    Aug 2, 2015 at 21:04
6
$\begingroup$

Here's one easy explanation. Linear regression finds the line that minimizes the sum-squared vertical distances to a line (assuming the predictor $x$ is on the horizontal axis and response $y$ is the vertical axis). When you treat $y$ as the predictor (leaving our axes fixed), linear regression will find a line that minimizes the sum-squared horizontal distances of your data to the line, typically resulting in a different line.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks. I was pretty sure I had just overlooked something relatively obvious. This appears to be the obvious thing I had overlooked. $\endgroup$
    – dsimcha
    Mar 23, 2010 at 22:35
  • 1
    $\begingroup$ In response to JBL's comment about perpendicular distances, this is actually done in practice; it goes under the name "orthogonal regression" and is closely related to principal component analysis. If you want to do this with a non-zero intercept, though, you already have to step up the math content of your problem (it becomes the problem of finding a 2D surface in 3D, or more precisely, the normal to it). $\endgroup$
    – Mike McCoy
    Mar 24, 2010 at 16:51
0
$\begingroup$

If you have a more-or-less circularly shaped cloud of data points with correlation 0, then the slopes should both be 0, not reciprocals of each other! You're trying to estimate the average y-value for a given x-value and vice-versa. With low correlations, the two lines should be nowhere near each other. The "reciprocal" argument makes sense only if they're both the same line, and that's only when the absolute value of the correlation is 1.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ The "perpendicular" idea is seen to make no sense if you consider a case where they x-values are in inches and the y-values is dollars, and you suddenly change from inches to feet. Which direction is considered "perpendicular" shouldn't depend on which units of measurement are used. $\endgroup$
    – Michael Hardy
    May 27, 2010 at 20:57
-1
$\begingroup$

I was taught that it is a property of correlation coefficient that $r$ the correlation of $X$ with $Y$ is the same as of $Y$ with $X$.

(from the course pdf file with notes):

LO 5. Note that correlation coefficient ($R$, also called Pearson's $R$) has the following properties:

  • the magnitude (absolute value) of the correlation coefficient measures the strength of the linear association between two numerical variables
  • the sign of the correlation coefficient indicates the direction of association
  • the correlation coefficient is always between -1 and 1, -1 indicating perfect negative linear association, +1 indicating perfect positive linear association, and 0 indicating no linear relationship
  • the correlation coefficient is unitless
  • since the correlation coefficient is unitless, it is not affected by changes in the center or scale of either variable (such as unit conversions)
  • the correlation of $X$ with $Y$ is the same as of $Y$ with $X$
  • the correlation coefficient is sensitive to outliers

But since then, I cannot find any other reference to this, that agrees with the symmetry thing. Maybe someone has some explanation.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.