# Variance of a weighted linear regression

$$Y_i$$ are independent random variables following a normal law of mean $$m_i = Ax_i + B$$ and variance $$V.$$

Let's take a sample $$y_i \sim Y_i.$$

I determine $$a$$ and $$b,$$ the weigthed least squares coefficients with weights $$w_i$$ of sum $$1.$$ I am interested in an unbiased estimator of variance $$V.$$

$$\sum w_i (y_i - a x_i - b)^2$$

is obviously biased but I don't manage to get anywhere close to a simple expression for an unbiased eatimate (In the case of the constant fit, it's fairly easier,see unbiased estimate of the variance of a weighted mean.)

Any ideas or references?

EDIT: for the unweighted regression, it's quite standard and a factor $$n / (n - 2)$$ is applied. But it won't work with weights (hint: take $$w_1 = 0.$$)

Rationale

I've been asked why I would need to assign different weights if all data points have the same variance.

I have two main cases from real life (physics):

• The relative variances are known but the normalisation is unknown. Thus the variances are $$\sigma_i^2 = \sigma^2 / w_i$$ with $$w_i$$ known but $$\sigma$$ remains to be determined.
• The data follow a linear law only locally, so I want to filter out distant data with some weight function such as $$w_i = \exp (-k^2 (x_i - x_0)^2)$$.

Solution

I managed to come up with a solution.

Using $$x_i' = x_i - \sum_j w_j x_j$$ the biased estimator for the variance $$\hat\sigma_\text{b}^2 = \sum_i w_i (y - a x_i - b)^2$$ can be written as $$\hat\sigma_\text{b}^2 = \sum_i w_i y_i^2 - \left(\sum_i w_i y_i\right)^2 - \frac{\left(\sum_i w_i x_i' y_i\right)^2}{\sum_i w_i x_i'^2}$$

To make the derivation easier, I will assume that the law I am trying to fit has $$A=B=0$$ so that $$E(y_i) = 0$$ and $$E(y_i y_j) = \delta_{ij} \sigma^2$$. With that in mind I can expand the squares into double sums, notice that indices $$i \ne j$$ cancel and finally find that $$\hat\sigma_\text{b}^2 = \sigma^2 - \left(\sum_i w_i^2 \right) \sigma^2 -\frac{\sum_i w_i^2 x_i'^2}{\sum_i w_i x_i'^2} \sigma^2$$

Thus I can write the unbiased estimate as $$\hat\sigma^2 = \frac{N}{N - \Delta N_\text{free}} \hat\sigma_\text{b}$$ where $$\Delta N_\text{free} = N \left[ \sum_i w_i^2 + \frac{\sum_i w_i^2 x_i'^2}{\sum_i w_i x_i'^2} \right]$$ is the loss of degrees of freedom. For equal weights ($$w_i = 1/N$$) it equals two, but will be larger than that for unequal weights.

Follow-up question

It seems simple enough that it must be somehow a well-known result. Any reference?

• Often one uses weights proportional to the reciprocals of the variances, since that minimizes the variance of the least-squares estimators of $a$ and $b.$ But you are assuming the variances are all the same. Is there are reason for using some other weights? Commented Sep 6 at 14:15

This is a standard fact in the theory of linear statistical models. See e.g. formulas (4.5) and (4.7) in Chapter 4, which present the following unbiased estimator of the variance: $$\hat\sigma^2=\frac1{n-2}\, \Big(\sum_{i=1}^n Y_i^2-n\bar Y^2-\frac{\big(\sum_{i=1}^n(x_i-\bar x)Y_i\big)^2}{\sum_{i=1}^n(x_i-\bar x)^2}\Big),$$ where $$\bar Y:=\frac1n\,\sum_{i=1}^n Y_i$$ and $$\bar x:=\frac1n\,\sum_{i=1}^n x_i$$.
(It is of course assumed here that $$n\ge3$$ and at least two of the $$x_i$$'s are distinct from each other.)
• What do you mean, precisely, by "in the weighted fit"? Why (and how) should an unbiased estimator of $V$ depend on your choice of estimators of $A$ and $B$? Also, using unequal weights given the same variance for all $i$ does not seem reasonable. Commented Sep 5 at 14:17
• @loqueelviento : I have commented on your reasons for unequal weights. As for $A$ and $B$, I did not say that you chose them. Instead, I said you chose (weighted) estimators of $A$ and $B$. These estimators will depend on your choice of the weights. I also said that there is no reason to tie estimation of $V$ to your choice of estimators for $A$ and $B$. Commented Sep 6 at 20:35