Let's say I have a linear regression model of the form $ y = B_x x + I_x + \epsilon $, where $B_x$ is the beta coefficient of the $x$ term, $I_x$ is the intercept term and $\epsilon$ is additive, normally distributed noise. If I have a dataset and perform linear regression, I get a value for $B_x$, which indicates the slope of the relationship.

If I swap the roles of the $x$ and $y$ data, and try to fit a model of $x = B_y y + I_y + \epsilon$, I would expect intuitively that $B_y = \frac{1}{B_x}$. A simple geometric argument can be made to show that swapping the roles of $x$ and $y$ shouldn't change the position of the regression line w.r.t. any data point, and from here it seems like simple algebra that if $y = Bx + I$ then $x = \frac{1}{B} y + \frac{I}{B}$.

Where is this reasoning wrong? Can someone explain to me why $B_x \neq \frac{1}{B_y}$, preferably without resorting to tons of linear algebra or direct derivation from the normal equation?