How does a statistical divergence change under a Lipschitz push-forward map?

Question

Let $\mu, \nu$ be two probability measures on a space $X$ (assume Polish space). $T: X \rightarrow Y$ is a Lipschitz-map that acts as a push-forward on these measures; let $\mu^\prime = T_{\#\mu}$ and $\nu^\prime = T_{\#\nu}$ be the resulting push-forward measures on the space $Y$.

Let $d(\mu, \nu)$ denote some distance function between measures. What can be said about how $d(\mu, \nu)$ and $d(\mu^\prime, \nu^\prime)$ relate to each other? Especially, how is the relationship a function of the Lipschitz constant of $T$?

Of course the relationship is likely to be a function of the exact nature of $d()$ (e.g., KL-divergence, or Total-variation or Wassterstein .. ). I suspect a lot must be known about this type of questions; what would be the right place to look at?

Answer 1

If $d(\mu,\nu)$ is taken to be the total variation metric, then Lipschitz and metric properties don't matter. This is due to a "data processing inequality" of sorts: applying a transformation can only make two distributions closer in TV. I'll illustrate this for discrete sets $X,Y$: $$ d(\mu',\nu') =\sum_{y\in Y}|\mu'(y)-\nu'(y)| =\sum_{y\in Y}|\sum_{x\in T^{-1}(y)}\mu(x)-\nu(x)| \le\sum_{x\in X}|\mu(x)-\nu(x)|=d(\mu,\nu). $$

For other metrics, the Lipschitz constant will matter. For example, take two densities $\mu,\nu$ on the real line and put $T(x)=a x$ for some $a>0$. Then $$ d_p(\mu',\nu'):=(\int |\mu'-\nu'|^p)^{1/p} = a^{(1-p)/p}d_p(\mu,\nu). $$ The last example is taken from Ch. 1 of https://www.szit.bme.hu/~gyorfi/nonpar_dens_en.html ; you may find the discussion there useful.

Answer 2

For the 1-Wasserstein metric $W_1$ on the Borel probability measures $\mathcal P(X)$ of a compact metric space $(X,d)$, recall the Kantorovich-Rubinstein formula:

$$ W_1(\mu,\nu) = \sup_{f\in\text{Lip}_1(X)} \mathbb E_{x\sim\mu}[f(x)] - \mathbb E_{x\sim\nu}[f(x)] = \sup_{f\in\text{Lip}_1(X)} \int_X f \,\mathrm d(\mu-\nu) $$

where $\text{Lip}_1(X)$ is the set of 1-Lipschitz $\mathbb R$-valued functions and $\mu,\nu\in\mathcal{P}(X)$.

Now assume that $g:X\to Y$ is $K$-Lipschitz and let $g_*\mu$ and $g_*\nu$ denote the push-forward measures. Note that if $f\in\text{Lip}_1(Y)$ then $\frac{1}{K}f\circ g\in \text{Lip}_1(X)$. Thus,

\begin{align*} W_1(g_*\mu,g_*\nu) &= \sup_{f\in\text{Lip}_1(X)} \int_X f \,\,\mathrm d(g_*\mu - g_*\nu) \\ &= \sup_{f\in\text{Lip}_1(X)} \int_X f \,\,\mathrm dg_*(\mu - \nu) \\ &= \sup_{f\in\text{Lip}_1(X)} \int_X f\circ g \,\,\mathrm d(\mu - \nu) \\ &= K \sup_{f\in\text{Lip}_1(X)} \int_X \frac{f\circ g}{K} \,\,\mathrm d(\mu - \nu) \\ &\leq K \sup_{h\in\text{Lip}_1(X) } \int_X h \,\,\mathrm d(\mu - \nu) \\ &= K\cdot W_1(\mu,\nu) \end{align*}