Bounding the sensitivity of a posterior mean to changes in a single data point

Question

There is a real-valued random variable $R$. Define a finite set of random variables ("data points") $$X_i = R + Z_i \; \text{for } i\in\{1,\ldots,n\},$$ where $Z_i$ are identically and independently distributed, mean-zero, and independent of $R$. The prior distributions of $R$ and the $Z_i$ are given; the support of each is the real line.

Define the conditional, or posterior, expectation of $R$ given particular realizations of the data $$\hat{R}(x_1,\ldots,x_n)= \mathbb{E}[R \mid X_1=x_1,\ldots,X_n=x_n]. $$ (This is the expectation of $R$ assessed by someone who sees the data points but not $R$ itself, and is a measurable function $\mathbb{R}^n \to \mathbb{R}$.) The question is: how much can an adversary with a given amount of manipulation power over a single data point move this estimate?

More precisely, fix a number $\Delta$. I am looking for a bound, which is useful as $n$ grows large, on $$M_n(\Delta)=\mathbb{E}[\hat{R}(X_1+\Delta,X_2,\ldots,X_n) - \hat{R}(X_1,X_2,\ldots,X_n)].$$ This expectation is an integral over all uncertainty in the model (i.e. in $R$ and the $X_i$), though I believe it should be possible to give a good bound even conditional on $R$.

We can take all random variables to be square-integrable if necessary, and make any other convenient assumptions.

A conjecture is that the manipulability is small in the sense that $M(\Delta) \to 0 $ as $n \to \infty$ and indeed $M_n'(\Delta) \to 0$ as well.

The conclusion may seem obvious because the posterior distribution of $R$ conditional on the data $X_1,\ldots,X_n$ converges to a point mass whose location is independent of the realization of $X_1$. But this does not readily imply a bound on the $L^1$ norm of the difference between $\hat{R}(X_1+\Delta,\ldots,X_n)$ and $R$, or the difference between $\hat{R}(X_1+\Delta,\ldots,X_n)$ and the unmanipulated estimate $\hat{R}(X_1,\ldots,X_n)$. It could be that the manipulation has a slowly decaying probability of achieving very large deviations in the estimate, so that it messes up the expectation.

Answer 1

There is no bound independent of $R$.

In what follows, I use my proposed notation, with $Y$ instead of $R$. Take \begin{align} Z &\sim N(0,1) \\ Y &\sim \text{even mix of } N(b,1) \text{ and } N(-b,1) \\ X &\sim \text{even mix of } N(b,\sqrt{2}) \text{ and }N(-b,\sqrt{2}) \end{align} So \begin{align} P(Z=z) &= \frac{1}{\sqrt{2\pi}\ \ }\,e^{-z^2/2} \\ P(Y=y) &= \frac{1}{2\sqrt{2\pi}}\left(e^{-(y-b)^2/2} + e^{-(y+b)^2/2}\right)\\ P(X=x) &= \frac{1}{\ 4\sqrt{\pi}\ }\left(e^{-(x-b)^2/4} + e^{-(x+b)^2/4}\right) \end{align} Suppose we have a single observation, namely $x$. Then \begin{align} P(Y'=y|X=x) &= \frac{P(x|y)P(y)}{P(x)}\\ &= \frac{\frac{1}{\sqrt{2\pi}\ \ }\,e^{-(x-y)^2/2}\frac{1}{2\sqrt{2\pi}}\left(e^{-(y-b)^2/2} + e^{-(y+b)^2/2}\right)} {\frac{1}{\ 4\sqrt{\pi}\ }\left(e^{-(x-b)^2/4} + e^{-(x+b)^2/4}\right)}\\ &= \frac{e^{-(x^2+b^2)/2}\left(e^{-y^2+by+xy} + e^{-y^2-by+xy}\right)} {\sqrt{\pi}\left(e^{-(x-b)^2/4} + e^{-(x+b)^2/4}\right)} \\ \\ E[Y'|X=x] &= \int_{y=-\infty}^\infty y\,P(Y'=y|X=x)\,dy\\ &= \frac{e^{-(x^2+b^2)/2}\left((x+b)e^{(x+b)^2/4} + (x-b)e^{(x-b)^2/4}\right)} {2\left(e^{-(x-b)^2/4} + e^{-(x+b)^2/4}\right)} \\ &= \frac{(x+b)e^{bx/2} + (x-b)e^{-bx/2}} {2\left(e^{bx/2} + e^{-bx/2}\right)} \\ \\ \frac{dE[Y'|X=x]}{dx}{\Large|}_{x=0} &= \frac{b^2+2}{4} \end{align} So the expectation of $Y'$ can be made to depend on $x$ with arbitrarily large sensitivity. Any bound on this sensitivity would likely be of the order of the variance of $Y$.