MathOverflow is a question and answer site for professional mathematicians. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Background James-Stein estimator and Stein's phenomenon, as described in Wikipedia are rather counterintuitive and amazing.

It is claimed that if one wants to estimate the mean $\Theta$ of
Gaussian distributed vector $ y$~$ N(\Theta, \sigma^2 Id)$, then the naive estimation - (i.e. just take $y$ as an estimation) is not good for size of vector greater or equal 3.

"Not good" means that (quote Wikipedia) "James–Stein estimator always achieves lower (Mean squared error (MSE) than the least squares estimator".

Question please clarify the sentence above. I wonder the following - usually when we calculate MSE we need some distribution on the estimated parameter $\Theta$ and averaging in MSE is taken over this distribution also.

So what distribution is assumed ? Or may be for ANY distribution it holds true ?


Stein's example is more general (quote Wikipedia):

Stein's example (or phenomenon or paradox), in decision theory and estimation theory, is the phenomenon that when three or more parameters are estimated simultaneously, there exist combined estimators more accurate on average (that is, having lower expected mean-squared error) than any method that handles the parameters separately. This is surprising since the parameters and the measurements might be totally unrelated.

A Paradox?

Popular articles have appeared hailing the James-Stein estimator a paradox; one should use the price of tea in China to obtain a better estimate of the chance of rain in Melbourne!

(Quote from Deane Yang's suggested page )

share|cite|improve this question
This seemed helpful to me:… – Deane Yang Apr 11 '12 at 9:30
@Deane Yang Thank You ! – Alexander Chervov Apr 11 '12 at 10:16

This has always bothered me. "One should use the price of tea in China to obtain a better estimate of the chance of rain in Melbourne" is not a good characterization at all. One should use the price of tea in China and the chance of rain in Melbourne to obtain a better estimate of the vector which includes both the average price of tea in China and the chance of rain in Melbourne. The Stein result only obtains if you care about a vector-valued parameter; that is the observations are assumed independent probabilistically but clearly interact with one another via the loss function being use.

The idea behind the quote is that you can hedge your bets on any given coordinate dimension by "shrinking" back towards the "global" mean (across all elements of the mean vector). But observe that the "shrinkage" need not in fact be towards the overall mean for the result to can shrink back towards any value at all and still get the result, which has to do with the definition of admissibility used. My favorite description of what is going on is here.

share|cite|improve this answer
In fact, I thought the link I provided made the same point. – Deane Yang Apr 11 '12 at 14:02
@Deane Yang, I'm sure it did, but I took the opportunity to rant :) – R Hahn Apr 11 '12 at 14:09
Yes, it's a good rant. I'm glad you made the same point explicitly. – Deane Yang Apr 11 '12 at 15:30

To some of the specific points raised in this question, here is a write up that actually shows the James Stein estimator in action. Specifically, there is R code you can use to verify it yourself for various distributions (Normal and Poisson). A key point to note is that the estimator does return closer "true" variables than MLE. Hope that helps

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.