I am trying to understand shrinkage, or the Stein phenomenon. As someone without a statistics background, the focus in most introductory presentations on normal distributions and squared error loss strikes me as somewhat arbitrary. Can the phenomenon be presented either more simply (using a different example) or more generally? Also, is there anything special about the number three? (Why does the phenomenon happen with three dimensions but not with fewer?)
Thinking about these issues, I tried to reconstruct something like the phenomenon in a simpler setting with just one variable. I think I must be missing something important because most presentations stress the importance of the dimensionality to the phenomenon, but I don't see where I have gone wrong.
Here is my example.
Let the parameter space consist of the even integers. The distribution corresponding to $\theta=n$ is the mean $n$ distribution that has half its weight on $n-1$ and half its weight on $n+1$. You will see a single draw from the distribution, i.e., an odd integer.
The obvious estimate of the parameter is the odd integer $x$ you see. How good an estimate that is (and, in particular, whether it is admissible) depends on the loss function we consider. Consider first absolute loss.
With this loss function, the expected loss is 1, and $x+\epsilon$ is an equally good estimator, for any fixed $\epsilon$ in $[-1,1]$. But suppose we let $\epsilon$ be a function of $x$. By making $\epsilon(x)$ a strictly decreasing function (that still stays within $[-1,1]$), we can beat the obvious estimator regardless of what $\theta$ is. Specifically, when $\theta=n$, if $n$ is non-zero, the improvement will be the difference between $\epsilon(n+1)$ and $\epsilon(n-1)$. The admissible estimators would appear to be those where $\lim_{x\to-\infty}\epsilon(x)=1$ and $\lim_{x\to\infty}\epsilon(x)=-1$.
Under a strictly convex loss function like squared loss, things are more complicated. The fixed $\epsilon$ estimator with $\epsilon=0$ now dominates all the other fixed $\epsilon$ estimators. Specifically, with squared loss, the expected loss of a fixed $\epsilon$ estimator is $1+\epsilon^2$. I originally thought that it might be possible to dominate this estimator by letting $\epsilon$ be a decreasing function, just as before. On further reflection, I realize that that doesn't seem to be possible unless we restrict the parameter space to the non-negative even integers. (But it would presumably be possible for a less severely curved but still strictly convex loss function.)
Is this the shrinkage phenomenon, or is it something else?
