Let's say we have two Gaussian distributions $\mathcal{N}(\mu_1, \sigma^2I_d)$ and $\mathcal{N}(\mu_2, \sigma^2I_d)$. We are trying to get estimators $\hat \mu_1, \hat \mu_2$ to minimize the following quantity, $$\mathbb{E}_{\hat \mu_1, \hat \mu_2}\left [\frac{\|\mu_1 - \hat \mu_1\|_2^2 + \|\mu_2 - \hat \mu_2\|_2^2}{2}\right],$$ i.e., the average $L_2$ distance to the actual means. If our estimators are unbiased, this is also the average variance of our estimators. Now assume we have $N$ samples from both distributions, i.e., total $2N$ samples. I am trying to understand the lowest we can get this error using these samples. For instance if I knew that $\mu_1\approx \mu_2$, I would perhaps use all the $2N$ samples and estimate $\frac{\mu_1 + \mu_2}{2}$. On the other hand, if I knew that the means are very far apart, the best thing to do (at least for unbiased estimators) is to use the respective MLE estimates for $\mu_1$ and $\mu_2$. I am wondering if there is a general answer to this question, which depends on the following quantity, $$\zeta_\star^2 := \left\|\frac{\mu_1 - \mu_2}{2}\right\|_2^2,$$, i.e., a theorem of alternative of sorts. More generally, when there are more than two distributions, say $M$, is there a standard way to prove variance lower bounds which depend on the following quantity, $$\zeta_\star^2 := \frac{1}{M}\sum_{m=1}^{M}\|\mu_m - \bar \mu\|_2^2,$$ where $\bar \mu = \frac{1}{M}\sum_{m=1}^{M}\mu_m$.

I looked into the standard technique for proving the lower bounds for Gaussian mean estimation (Le Cam's method and Cramer-Rao lower bounds), but I could not extend them to the case when we have more than one mean. I asked ChatGPT what to do, and it gave several wrong answers but at least suggested that Bhatacharya Coefficient might be relevant to answering this question.

I would appreciate it if you have any pointers. This is not a HW question. Also, this is my first post, so I apologize for missing any guidelines.

**Edit 1**

After reading a bit on James-Stein Estimator and following the comments on the original question, it makes sense to clarify my problem and ask a more rigorous question. We have samples of size $N$ from $M$ different Gaussian distributions, each with mean $\mu_m$ and a known co-variance $\sigma^2I_d$. We want to get estimators $\hat\mu_1, \dots, \hat \mu_M$ to minimize the average mean squared error (AMSE): $$AMSE(\hat\mu_1, \dots, \hat \mu_M) := \mathbb{E}\left[\frac{1}{M}\sum_{m\in[M]}\|\hat \mu_m - \mu_m\|_2^2\right],$$ where the expectation is w.r.t.,

- sampling from these distributions, i.e., obtaining $X_{m,1}, \dots, X_{m,N}\sim \mathcal{N}(\mu, \sigma^2I_d)$ for all $m\in[M]$; and
- potential randomness in our estimators $\hat\mu_1, \dots, \hat \mu_M$.

We know that the simple sample mean estimators $\hat\mu_m^{MLE} := \frac{1}{N}\sum_{i\in[N]}X_{m,i},\ \forall m\in[M]$ are inadmissible (i.e., are dominated by other estimators for some value of the means) when $M\geq 3$. This is due to the famous Stein's example. One can calculate the MSE of this estimator as follows, $$AMSE(\hat\mu_1^{MLE}, \dots, \hat\mu_M^{MLE}) = \frac{\sigma^2 d}{N}.$$ While inadmissible, this is optimal for unbiased estimators due to the Cramer-Rao Lower bound. Now, I am wondering if there are biased estimators that obtain better AMSE. In particular, consider the estimator which averages the samples across all the distributions, i.e., $$\hat\mu_1 = \dots = \hat \mu_M = \hat \mu^{ALL} = \frac{1}{MN}\sum_{m\in[M], i\in[N]} X_{m,i}.$$ This estimator has the following AMSE, $$AMSE(\hat \mu^{ALL}, \dots, \hat \mu^{ALL}) = \frac{\sigma^2d}{MN} + \frac{1}{M}\sum_{m=1}^{M}\|\mu_m - \bar \mu\|_2^2,$$ where $\bar \mu := \frac{1}{M}\sum_{m=1}^{M}\mu_m$. Now let's assume $\|\mu_m\|_2 = B$ for all $m\in[M]$, so we even know the scale of the true parameter (which is the key quantity we vary to check if an estimator is admissible). My question is whether there is an approach to study the smallest the AMSE can be as a function of $$\zeta_\star^2 := \frac{1}{M}\sum_{m=1}^{M}\|\mu_m - \bar \mu\|_2^2,$$ i.e., the "heterogeneity" of the distributions. One can see that when $\zeta_\star = 0$, the ideal thing is to use the estimator $\hat\mu^{ALL}$, while if $\zeta_\star$ is large, it "intuitively" seems like the best estimator is some admissible estimator for the problem such as Jason-Stein applied for the concatenated vector of means of dimension $dM$. Is it possible to say something intelligent in between these regimes? If I squint a little (a lot, perhaps), I can see a trade-off here akin to the bias-variance trade-off. That's what I am trying to understand.

Thanks for any pointers in advance!

3more comments