Take the 2-minute tour ×
MathOverflow is a question and answer site for professional mathematicians. It's 100% free, no registration required.

I have a follow-up question to this one:

unbiased estimate of the variance of a weighted mean

Specifically, how do I generalise the result given here (and on Wikipedia) for the unbiased sample estimate of the variance of a weighted population to the case where the weights are not normalised to 1? (or equivalently are not in the standard simplex, as in the previous question's answer derivation)

I'm not sure how much of the previous answer relied on the weights being in the unit simplex, but it's clear that the given answer contains denominator terms like $1 - \sum_i w_i^2$ which aren't going to be nice if $\sum_i w_i^2 > 1$! Maybe there's a simple ansatz for modification to unnormalized weights, but it's not obvious to me which to choose!

Thanks!

Andy

share|improve this question
add comment

1 Answer

up vote 1 down vote accepted

Hi, Rather long after your question, but it can be done directly in the same way Matus did it, or you can simply use the following:

Matus assumed weights Wi which sum to 1. Suppose you have weights Ui, and write V1 = sum of the Ui, and V2 = sum of the Ui^2, consistent with the Wikipedia entry for weighted sample variance. Then we can put Wi = Ui/V1.

Now, look at the factor 1 / (1 - sum(Wi^2)), replace the Wi with Ui/V1, multiply top and bottom lines by V1^2 and - voila! - you get V1^2 / { V1^2 - V2 } .

However, like Matus, I'm wondering when you would ever use such a "weighted sample variance" - see my question as a response to the original post.

I suspect there is much confusion over the different reasons for weighting.

Kathy

share|improve this answer
    
Hi Kathy... thanks for the response, and sorry that mine has also taken a long time to get around to. For the particular problem that we had, I believe that we found a suitable solution some time ago by use of an "effective N", computed as $(\sum W_i)^2 / \sum W_i^2$. I'd have to have a bit of a think to see if that is equivalent to the substitutions that you propose. Some histogramming code that implements this scheme (and which produces reasonable-looking results) is here: projects.hepforge.org/rivet/trac/browser/trunk/include/LWH/… –  andybuckley Jun 2 '11 at 7:14
    
I concur that there seem to be quite differing opinions on what weights are for. In my case, they come from samplers used in physics code: to generate adequate statistical coverage for regions of the sampling phase space which are physically suppressed, the sampled function is multiplied by an enhancement function. The raw distributions are then unphysical, so sampled points need to be down-weighted by the relevant enhancement factor when computing observables: this weight needs to be propagated into the calculation of uncertainties. Hope that clarifies a bit. Thanks again :) –  andybuckley Jun 2 '11 at 7:19
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.