## Unbiased estimate of the variance of an *unnormalised* weighted mean

I have a follow-up question to this one:

http://mathoverflow.net/questions/11803/unbiased-estimate-of-the-variance-of-a-weighted-mean

Specifically, how do I generalise the result given here (and on Wikipedia) for the unbiased sample estimate of the variance of a weighted population to the case where the weights are not normalised to 1? (or equivalently are not in the standard simplex, as in the previous question's answer derivation)

I'm not sure how much of the previous answer relied on the weights being in the unit simplex, but it's clear that the given answer contains denominator terms like $1 - \sum_i w_i^2$ which aren't going to be nice if $\sum_i w_i^2 > 1$! Maybe there's a simple ansatz for modification to unnormalized weights, but it's not obvious to me which to choose!

Thanks!

Andy

-

Hi, Rather long after your question, but it can be done directly in the same way Matus did it, or you can simply use the following:

Matus assumed weights Wi which sum to 1. Suppose you have weights Ui, and write V1 = sum of the Ui, and V2 = sum of the Ui^2, consistent with the Wikipedia entry for weighted sample variance. Then we can put Wi = Ui/V1.

Now, look at the factor 1 / (1 - sum(Wi^2)), replace the Wi with Ui/V1, multiply top and bottom lines by V1^2 and - voila! - you get V1^2 / { V1^2 - V2 } .

However, like Matus, I'm wondering when you would ever use such a "weighted sample variance" - see my question as a response to the original post.

I suspect there is much confusion over the different reasons for weighting.

Kathy

-
 Hi Kathy... thanks for the response, and sorry that mine has also taken a long time to get around to. For the particular problem that we had, I believe that we found a suitable solution some time ago by use of an "effective N", computed as $(\sum W_i)^2 / \sum W_i^2$. I'd have to have a bit of a think to see if that is equivalent to the substitutions that you propose. Some histogramming code that implements this scheme (and which produces reasonable-looking results) is here: projects.hepforge.org/rivet/trac/browser/trunk/… – Andy Buckley Jun 2 2011 at 7:14 I concur that there seem to be quite differing opinions on what weights are for. In my case, they come from samplers used in physics code: to generate adequate statistical coverage for regions of the sampling phase space which are physically suppressed, the sampled function is multiplied by an enhancement function. The raw distributions are then unphysical, so sampled points need to be down-weighted by the relevant enhancement factor when computing observables: this weight needs to be propagated into the calculation of uncertainties. Hope that clarifies a bit. Thanks again :) – Andy Buckley Jun 2 2011 at 7:19