(I hope this is not too basic)

I basically have a set of data from the same underlying distribution (which I would like to estimate), but I only have available the mean and N from partitions of the data. I can estimate the mean using a weighted average, and, I thought, the sample variance using weighted variance (e.g: https://stat.ethz.ch/pipermail/r-help/2008-July/168762.html)

However, I can't make much sense of the results. And I suspect I'm misunderstanding how/what weighted sample average should work.

Edit: From a set of (unobserved) data points all drawn from some distribution with some $\mu$ and $\sigma$, I can observe only the average and cardinality for partitions of the data. I.e. my observed values $X_i$ are each the average of some partition of size $N_i$, and thus have, $\mu_i=\mu$ and a $\sigma_i=\sigma/\sqrt{N_i}$.

Question: how can I estimate $\mu$ and $\sigma$ from the observed $X_i$s? Note that I can't observe the variance (or any other parameters) for the partitions.

The goal is to identify partitions that do not fit the expected distribution, i.e. identify partitions where $X_i$ is too many $\sigma_i$s from $\mu$.

  • $\begingroup$ As far as I understood, you have a set of, let's say $M$ partitions of different data with cardinality $N_i$, mean $\mu_i$ mean and variance $\sigma^2_i$ for all $i = 1, \ldots , M$. Is it right? $\endgroup$ – mikitov May 6 '11 at 13:33
  • $\begingroup$ it does appear too basic for this site, although like mikitov I'm not 100% sure I understand what you mean. Did you consider asking your question on stats.stackexchange.com $\endgroup$ – Thierry Zell May 6 '11 at 13:39
  • $\begingroup$ Like Thierry I don't quite understand what exactly you are asking for. For one possible interpretation ("you are looking for a numerical formula") I'd recommend you to read infoserve.sandia.gov/sand_doc/2008/086212.pdf and for another possible interpretation ("you are speaking about quantization") to read statistik.uni-dortmund.de/fileadmin/user_upload/Lehrstuehle/… $\endgroup$ – Someone May 6 '11 at 14:24
  • $\begingroup$ I didn't know about stats.stackexchange.com, but I can try there. I don't know how to clarify it or explain better - I looked at the links but it is not quanization, and I don't understand the other link enough to see any relevance. $\endgroup$ – Ketil May 6 '11 at 20:26
  • $\begingroup$ Maybe the same question as this?stats.stackexchange.com/questions/10441/… $\endgroup$ – Ketil May 6 '11 at 20:30

Note: New answer.

If you don't know the variances $Var X_1,...,VarX_m$, you have to make parametric assumptions about the distribution of the data. For example, under the assumption of a Poisson distribution, the the maximum likelihood estimator of the variance would simply be the aggregated mean $EX$. Under the assumption of the normal distribution, the intuitive answer

$$VarX = (1/m)\sum_i N_i (EX_i-\mu)^2$$ is the maximum likelihood estimator.

Assuming your distirbution is normal $(\mu, \sigma^2)$, then the maximum likelihood estimate of $\mu$ is the aggregated mean $(1/N)\sum_i N_i EX_i$. The profile likelihood as a function of $\sigma^2$ is then

$$L(\sigma^2) = \prod_i \frac{1}{\sqrt{2\pi\sigma^2/N_i}} exp\left(\frac{-(EX_i - \mu)^2}{2\sigma^2/N_i}\right).$$

The maximum likelihood estimate of $\sigma^2$ is the maximizing argument of the above. For computational convenience we take the logarithm of the likelihood as it is easier to maximize.

$$\log L(\sigma^2) = \sum_i -\frac{1}{2}\log(2\pi/N_i) - \frac{1}{2}\log(\sigma^2) - \frac{(EX_i - \mu)^2}{2\sigma^2/N_i}$$

Setting the derivative to zero,

$$0=\frac{\partial L(\sigma^2)}{\partial \sigma^2} = \sum_i -\frac{1}{2\sigma^2} + \frac{(EX_i-\mu)^2}{2/N_i}\frac{1}{\sigma^4}$$

multiplying by $\sigma^4$,

$$ 0 = \sum_i -\frac{1}{2}\sigma^2 + \frac{N_i(EX_i-\mu)^2}{2}$$

which yields the maximum likelihood estimator

$$\sigma^2 = (1/m)\sum_i N_i (EX_i-\mu)^2.$$

For general parametric $f(\theta)$ you need the densities of the convolutions $f * \cdots * f(\theta)$, which makes the problem much more difficult outside of a few special parametric families. However if all of your $N_i$ are large, you can use the normal distribution as an approximation by the Central Limit Theorem.


@charles: You're misusing the word "sample" in the usual way; you should say two samples of sizes $N_1$ and $N_2$.

However (and this is the occasion for making this an "answer" rather than a comment), I think a conceptually more efficient way to think about this is to recall that $\text{var}(X) = E(\text{var}(X\mid Y)) + \text{var}(E(X \mid Y))$. I.e. the total variance is the explained variance plus the unexplained variance. See Law of total variance.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.