MathOverflow is a question and answer site for professional mathematicians. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a couple questions on the same topic:

1) Does ordinary linear least squares have a generally defined confidence interval. I know that you can determine the confidence interval of each term, $\hat{a}+\hat{b}x$ as follows:

$\hat{a}\pm t_{\alpha/2,n-2}\sqrt{(1/n+\hat{x}^2/S_{xx})MSE}$

$\hat{b}\pm t_{\alpha/2,n-2}\sqrt{MSE/S_{xx}}$

Is there a combination of these two terms that yields an overall confidence interval of the least squares regression?

2) Secondly, if a general confidence interval is defined for the linear regression, can it be generalized into higher degree regressions; $\hat{z_0}+\hat{z_1}x + \hat{z_2}x^2+...+\hat{z_n}x^n$?

I'm programming some regression classes and I'm trying to design a regression interface. I've managed to find confidence interval equations for more complicated regression techniques, but the simplest technique is giving me the most problems.

share|cite|improve this question

You seem to be suffering the usual confusion about what "linear regression" means. Nonlinearity in the $x$ values does not mean you're doing nonlinear regression. If you're talking about fitting the sort of polynomial you're talking about above under the simplest conventional assumptions, that's linear regression, notwithstanding the fact that you're fitting a curve rather than a line.

You're also using the lower-case $n$ to denote two different things in the same problem! Confusing at best!

Think about this: $$ Y = X\beta + \varepsilon $$ where $Y$ is an $n\times1$ vector, $X$ is an $n\times k$ matrix with (typically) $k \ll n$, and $\varepsilon$ is an $n\times 1$ random vector of "errors" that is normally distributed with expected value the $n\times 1$ column vector of $0$s and variance $\sigma^2$ times the $n\times n$ identity matrix. You can observe $X$ and $Y$ and you want to estimate $\beta$, which is of course a $k\times 1$ fixed but unobservable vector.

In the case where $Y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \cdots + \beta_{k-1} x_i^{k-1} + \varepsilon_i$, we have $$ X= \begin{bmatrix} 1 & x_1 & x_1^2 & \cdots & x_1^{k-1} \\ \vdots & \vdots & \vdots & & \vdots \\ 1 & x_n & x_n^2 & \cdots & x_n^{k-1} \end{bmatrix} $$ Then the least-squares estimate of $\beta$ is $\hat\beta = (X^TX)^{-1}X^T Y$. The vector of fitted values of $Y$ is $\hat Y = HY = X(X^TX)^{-1}X^TY$. The "hat matrix" $H$ is an $n\times n$ symmetrix idempotent matrix of rank $k \ll n$. It's the orthogonal projection onto the column space of $X$. The vector of residuals is $\hat\varepsilon = Y - \hat Y = (I - H)Y$. This matrix $I-H$ represents the complementary orthogonal projection.

Now remember: If $A$ is an $m\times n$ fixed matrix and $B$ is an $\ell\times n$ fixed matrix then $E(AY) = A E(Y)$, $\operatorname{var}(AY) = A(\operatorname{var}(Y))A^T$, and $\operatorname{cov}(AY,BY) = A(\operatorname{cov}(Y,Y)B^T$.

Consequently we have $$ \hat\beta \sim N_k (\beta, \sigma^2 (X^TX)^{-1}), $$ (a $k$-dimensional normal distribution) $$ \hat\varepsilon \sim N_n(0, \sigma^2(I - H)) $$ (an $n$-dimensional normal distribution with a variance of small rank, $k$), and $$ \operatorname{cov}(\hat\beta, \hat\varepsilon) = 0 $$ (a $k\times n$ matrix of zeros).

Therefore the distribution of the sum of squares of observed residuals (not unboservable errors) is given by $$ \frac{\|\hat\varepsilon\|^2}{\sigma^2} = \frac{\|(I-H)Y\|^2}{\sigma^2} \sim \chi^2_{n-k}, $$ a chi-square distribution with $n-k$ degrees of freedom, and this random vector is actually independent of the least-squares estimates $\hat\beta$, which is normally distributed.

Now if you think about the definition of the Student's t-distribution, you can derive various confidence intervals.

share|cite|improve this answer

Rather than a confidence interval, people use an F-test on the sum of squares. There's a description on Wikipedia. This generalizes to multiple regression, including polynomial regression.

In multiple regression, people look at two things: the t statistic for each individual coefficient, and the F statistic for the sum of squares. You can also define a confidence ellipsoid, which is like a confidence interval for all of the coefficients simultaneously. Wikipedia talks about it a little bit here.

share|cite|improve this answer

If you mean a "confidence interval" for the coefficients you're estimating, that's a point in a several-dimensional space, so you're probably looking for an ellipsoid rather than an interval. In the traditional way this is treated, the $x$ values are fixed and the $y$ values are random. The usual textbook confidence intervals are based on assumptions such as errors that are independent, that have expectation 0, and that all have the same variance, and that are normally distributed. If that is the case, then the whole vector of estimated coefficients depends linearly on the vector of $y$ values, and non-linearly in the vector of $x$ values. The random vector of estimates therefore depends linearly on that which is random. Its distribution is therefore easily determined. Then have the usual stuff about the distribution of residuals (as opposed to errors) and Student's t-distribution. Doesn't the book you got the formulas you quote have anything on this? If not, it's a cookbook for people who need to use math without doing math. What book is it?

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.