$Y_i$ are independent random variables following a normal law of mean $m_i = Ax_i + B$ and variance $V.$
Let's take a sample $y_i \sim Y_i.$
I determine $a$ and $b,$ the weigthed least squares coefficients with weights $w_i$ of sum $1.$ I am interested in an unbiased estimator of variance $V.$
$$\sum w_i (y_i - a x_i - b)^2$$
is obviously biased but I don't manage to get anywhere close to a simple expression for an unbiased eatimate (In the case of the constant fit, it's fairly easier,see unbiased estimate of the variance of a weighted mean.)
Any ideas or references?
EDIT: for the unweighted regression, it's quite standard and a factor $n / (n - 2)$ is applied. But it won't work with weights (hint: take $w_1 = 0.$)
Rationale
I've been asked why I would need to assign different weights if all data points have the same variance.
I have two main cases from real life (physics):
- The relative variances are known but the normalisation is unknown. Thus the variances are $\sigma_i^2 = \sigma^2 / w_i$ with $w_i$ known but $\sigma$ remains to be determined.
- The data follow a linear law only locally, so I want to filter out distant data with some weight function such as $w_i = \exp (-k^2 (x_i - x_0)^2)$.
Solution
I managed to come up with a solution.
Using $$x_i' = x_i - \sum_j w_j x_j$$ the biased estimator for the variance $$\hat\sigma_\text{b}^2 = \sum_i w_i (y - a x_i - b)^2$$ can be written as $$\hat\sigma_\text{b}^2 = \sum_i w_i y_i^2 - \left(\sum_i w_i y_i\right)^2 - \frac{\left(\sum_i w_i x_i' y_i\right)^2}{\sum_i w_i x_i'^2} $$
To make the derivation easier, I will assume that the law I am trying to fit has $A=B=0$ so that $E(y_i) = 0$ and $E(y_i y_j) = \delta_{ij} \sigma^2$. With that in mind I can expand the squares into double sums, notice that indices $i \ne j$ cancel and finally find that $$\hat\sigma_\text{b}^2 = \sigma^2 - \left(\sum_i w_i^2 \right) \sigma^2 -\frac{\sum_i w_i^2 x_i'^2}{\sum_i w_i x_i'^2} \sigma^2$$
Thus I can write the unbiased estimate as $$\hat\sigma^2 = \frac{N}{N - \Delta N_\text{free}} \hat\sigma_\text{b}$$ where $$\Delta N_\text{free} = N \left[ \sum_i w_i^2 + \frac{\sum_i w_i^2 x_i'^2}{\sum_i w_i x_i'^2} \right]$$ is the loss of degrees of freedom. For equal weights ($w_i = 1/N$) it equals two, but will be larger than that for unequal weights.
Follow-up question
It seems simple enough that it must be somehow a well-known result. Any reference?
