1
$\begingroup$

I have a follow-up question to this one:

unbiased estimate of the variance of a weighted mean

Specifically, how do I generalise the result given here (and on Wikipedia) for the unbiased sample estimate of the variance of a weighted population to the case where the weights are not normalised to 1? (or equivalently are not in the standard simplex, as in the previous question's answer derivation)

I'm not sure how much of the previous answer relied on the weights being in the unit simplex, but it's clear that the given answer contains denominator terms like $1 - \sum_i w_i^2$ which aren't going to be nice if $\sum_i w_i^2 > 1$! Maybe there's a simple ansatz for modification to unnormalized weights, but it's not obvious to me which to choose!

Thanks!

Andy

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Hi, Rather long after your question, but it can be done directly in the same way Matus did it, or you can simply use the following:

Matus assumed weights $W_i$ which sum to $1$. Suppose you have weights Ui, and write $V_1 = \sum U_i$, and $V_2 = \sum U_i^2$, consistent with the Wikipedia entry for weighted sample variance. Then we can put $\displaystyle W_i = \frac{U_i}{V_1}$.

Now, look at the factor $\displaystyle \frac{1} {(1 - \sum W_i^2)}$, replace the $W_i$ with $\displaystyle\frac{U_i}{V_1}$, multiply top and bottom lines by $V_1^2$ and - voila! - you get $\displaystyle \frac{V_1^2}{ V_1^2 - V_2 }$ .

However, like Matus, I'm wondering when you would ever use such a "weighted sample variance" - see my question as a response to the original post.

I suspect there is much confusion over the different reasons for weighting.

Kathy

Improve this answer
$\endgroup$
2
  • $\begingroup$ Hi Kathy... thanks for the response, and sorry that mine has also taken a long time to get around to. For the particular problem that we had, I believe that we found a suitable solution some time ago by use of an "effective N", computed as $(\sum W_i)^2 / \sum W_i^2$. I'd have to have a bit of a think to see if that is equivalent to the substitutions that you propose. Some histogramming code that implements this scheme (and which produces reasonable-looking results) is here: projects.hepforge.org/rivet/trac/browser/trunk/include/LWH/… $\endgroup$
    – andybuckley
    Commented Jun 2, 2011 at 7:14
  • $\begingroup$ I concur that there seem to be quite differing opinions on what weights are for. In my case, they come from samplers used in physics code: to generate adequate statistical coverage for regions of the sampling phase space which are physically suppressed, the sampled function is multiplied by an enhancement function. The raw distributions are then unphysical, so sampled points need to be down-weighted by the relevant enhancement factor when computing observables: this weight needs to be propagated into the calculation of uncertainties. Hope that clarifies a bit. Thanks again :) $\endgroup$
    – andybuckley
    Commented Jun 2, 2011 at 7:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .