Assume that $\theta$ is in a convex body $K \in \mathbb{R}^n$ and we observe $y = \theta + z$, where $z$ is a noise term (following, say, the normal distribution). Consider an estimator of $\theta$ by projecting $y$ onto $K$ and denote it by $\hat\theta_K$. How does the risk $\mathbb{E} \|\hat\theta_K - \theta\|^2_2$ depend on $K$? Is it true that $$\mathbb{E} \|\hat\theta_K - \theta\|^2_2 \le \mathbb{E} \|\hat\theta_{K'} - \theta\|^2_2$$ if $K'$ is a superset of $K$?
Thanks!
$\newcommand\th\theta\newcommand\R{\mathbb R}$The answer is no in general if the noise term is allowed to have, say, any distribution symmetric about the origin with finite second moments.
Indeed, let e.g. $n=2$, $$K':=[-40,10]\times[-18,3],$$ $$K:=\{(x_1,x_2)\in K'\colon\, 2x_1+x_2-2\le0\},$$ and $\th:=(0,0)$, so that $\th\in K\subset K'$. Let $Y$ be a random point in $\R^2$ such that $P(Y=y)=1/2=P(Y=-y)$, where $y:=(40,-3)$, so that $-y\in K\subset K'$. Then for the respective projections $\hat\th_K$ and $\hat\th_{K'}$ of $Y$ onto $K$ and $K'$ we have $$P(\hat\th_K=(10, -18))=1/2=P(\hat\th_K=-y),$$ $$P(\hat\th_{K'}=(10, -3))=1/2=P(\hat\th_{K'}=-y),$$ which implies $$E\|\hat\th_K-\th\|^2_2=\frac{2033}{2}>859=E\|\hat\th_{K'}-\th\|^2_2,$$ so that the inequality in question fails to hold.
This counterexample is illustrated by the following picture, showing the rectangle $K'$, the darker region $K$, $\th=(0,0)$ (red), points $\pm y$ (black), and the projections $(10,-18)$ and $(10,-3)$ of the point $y$ onto $K$ (green) and onto $K'$ (blue).
It remains unclear what happens if the noise has a centered Gaussian distribution.
