Variance of a weighted linear regression

Question

$Y_i$ are independent random variables following a normal law of mean $m_i = Ax_i + B$ and variance $V.$

Let's take a sample $y_i \sim Y_i.$

I determine $a$ and $b,$ the weigthed least squares coefficients with weights $w_i$ of sum $1.$ I am interested in an unbiased estimator of variance $V.$

$$\sum w_i (y_i - a x_i - b)^2$$

is obviously biased but I don't manage to get anywhere close to a simple expression for an unbiased eatimate (In the case of the constant fit, it's fairly easier,see unbiased estimate of the variance of a weighted mean.)

Any ideas or references?

EDIT: for the unweighted regression, it's quite standard and a factor $n / (n - 2)$ is applied. But it won't work with weights (hint: take $w_1 = 0.$)

---

Rationale

I've been asked why I would need to assign different weights if all data points have the same variance.

I have two main cases from real life (physics):

• The relative variances are known but the normalisation is unknown. Thus the variances are $\sigma_i^2 = \sigma^2 / w_i$ with $w_i$ known but $\sigma$ remains to be determined.
• The data follow a linear law only locally, so I want to filter out distant data with some weight function such as $w_i = \exp (-k^2 (x_i - x_0)^2)$.

Solution

I managed to come up with a solution.

Using $$x_i' = x_i - \sum_j w_j x_j$$ the biased estimator for the variance $$\hat\sigma_\text{b}^2 = \sum_i w_i (y - a x_i - b)^2$$ can be written as $$\hat\sigma_\text{b}^2 = \sum_i w_i y_i^2 - \left(\sum_i w_i y_i\right)^2 - \frac{\left(\sum_i w_i x_i' y_i\right)^2}{\sum_i w_i x_i'^2} $$

To make the derivation easier, I will assume that the law I am trying to fit has $A=B=0$ so that $E(y_i) = 0$ and $E(y_i y_j) = \delta_{ij} \sigma^2$. With that in mind I can expand the squares into double sums, notice that indices $i \ne j$ cancel and finally find that $$\hat\sigma_\text{b}^2 = \sigma^2 - \left(\sum_i w_i^2 \right) \sigma^2 -\frac{\sum_i w_i^2 x_i'^2}{\sum_i w_i x_i'^2} \sigma^2$$

Thus I can write the unbiased estimate as $$\hat\sigma^2 = \frac{N}{N - \Delta N_\text{free}} \hat\sigma_\text{b}$$ where $$\Delta N_\text{free} = N \left[ \sum_i w_i^2 + \frac{\sum_i w_i^2 x_i'^2}{\sum_i w_i x_i'^2} \right]$$ is the loss of degrees of freedom. For equal weights ($w_i = 1/N$) it equals two, but will be larger than that for unequal weights.

Follow-up question

It seems simple enough that it must be somehow a well-known result. Any reference?

Answer 1

This is a standard fact in the theory of linear statistical models. See e.g. formulas (4.5) and (4.7) in Chapter 4, which present the following unbiased estimator of the variance: $$\hat\sigma^2=\frac1{n-2}\, \Big(\sum_{i=1}^n Y_i^2-n\bar Y^2-\frac{\big(\sum_{i=1}^n(x_i-\bar x)Y_i\big)^2}{\sum_{i=1}^n(x_i-\bar x)^2}\Big),$$ where $\bar Y:=\frac1n\,\sum_{i=1}^n Y_i$ and $\bar x:=\frac1n\,\sum_{i=1}^n x_i$.

(It is of course assumed here that $n\ge3$ and at least two of the $x_i$'s are distinct from each other.)