Parameterization of exponential family Let $\{\mathbb{P}_{\theta}\}_{\theta}$ be an exponential family of probability measures, all with finite mean.  Under what conditions is the parameterization map $\theta\mapsto \mathbb{P}_{\theta}$ continuous with respect to the Wasserstein-$1$ distance?
 A: $\newcommand\om\omega\newcommand\Om\Omega\newcommand\ep\epsilon\newcommand\R{\mathbb R}$Let $(P_t)_{t\in T}$ be an exponential family over a separable complete metric space $(X,d)$, where $T$ is an open subset of $\mathbb R^k$ and $t=(t_1,\dots,t_k)$ is a natural parameter. So, for all $t\in T$, the density $p_t$ of $P_t$ with respect to Borel measure $\mu$ on $X$ is given by the formula
\begin{equation*}
    p_t(x)=c(t)\exp\{t\cdot U(x)\}\tag{1}
\end{equation*}
for all $x\in X$, where
$U\colon X\to\mathbb R^k$ is a Borel-measurable function, $\cdot$ denotes the dot product on $\mathbb R^k$,
\begin{equation*}
    c(t):=1/I(t),\quad I(t):=\int_X\mu(dx)\exp\{t\cdot U(x)\}\in(0,\infty). \tag{2} 
\end{equation*}
Assume that, for some (and hence for all) $x_0\in X$, the "first moment"
\begin{equation}
    m_t(x_0):=\int_X P_t(dx)d(x_0,x)=\int_X \mu(dx)p_t(x)d(x_0,x)<\infty\tag{2a}
\end{equation}
for all $t\in T$.
Then the parameterization map $U\ni t\mapsto P_t$ is continuous with respect to the Wasserstein-1 distance $W_1$.
Indeed, by the penultimate paragraph of the introduction section, if suffices to show that for each $t\in T$
\begin{equation*}
    \int_X \mu(dx)p_s(x)f(x)\to \int_X \mu(dx)p_t(x)f(x)\tag{3}
\end{equation*}
as $s\to t$, where $f\colon X\to\mathbb R$ such that
\begin{equation*}
C:=\sup_{x\in X}|f(x)|/(1+d(x_0,x))<\infty  \tag{3a}
\end{equation*}
(for some (and hence for all) $x_0\in X$).
To verify (3), take any $t=(t_1,\dots,t_k)$ in the open set $T$. Take any real $\ep>0$ such that $t+\ep\om\in T$ for all $\om=(\om_1,\dots,\om_k)\in\Om:=\{-1,1\}^k$.
Take then any $s\in T$ such that $\max_{j=1}^k|s_j- t_j|\le\ep$ and any $u\in\R^k$. Then
\begin{align*}\exp\{s\cdot u\}/\exp\{t\cdot u\}&=\exp\{(s-t)\cdot u\} \\ 
&\le\max_{\om\in\Om}\exp\{\ep\om\cdot u\} \\  
&\le\sum_{\om\in\Om}\exp\{\ep\om\cdot u\};
\end{align*}
the first inequality in the above display follows because the function $\exp$ is convex and the condition $\max_{j=1}^k|s_j- t_j|\le\ep$ implies that the point $s-t$ is in the convex hull of the set $\{\ep\om\colon\om\in\Om\}$. So, for all $x\in X$
\begin{equation*}
    \exp\{s\cdot U(x)\}\le g(x):=\sum_{\om\in\Om}\exp\{( t+\ep\om)\cdot U(x)\}. \tag{5}
\end{equation*}
Moreover, by (2),
\begin{equation*}
 \int_X\mu(dx)g(x)=\sum_{\om\in\Om}I(t+\ep\om)<\infty. 
\end{equation*}
So, by the dominated convergence theorem, $I(s)\to I(t)$ and hence $c(s)\to c(t)$ as $s\to t$. So, choosing $\ep$ small enough, we have
\begin{equation*}
    c(s)\le 2c(t). \tag{6}
\end{equation*}
In view of (1) and (6), inequality (5) yields
\begin{equation*}
    p_s(x)\le c(s)\sum_{\om\in\Om}\frac{p_{t+\ep\om}(x)}{c(t+\ep\om)}
    \le K\sum_{\om\in\Om}p_{t+\ep\om}(x),
\end{equation*}
where
\begin{equation*}
    K:=2c(t)\max_{\om\in\Om}\frac1{c(t+\ep\om)}<\infty. 
\end{equation*}
So, by (3a),
\begin{equation*}
    p_s(x)|f(x)|\le h(x):=KC\sum_{\om\in\Om}p_{t+\ep\om}(x)(1+d(x_0,x)).  
\end{equation*}
Moreover, by (2a),
\begin{equation*}
    \int_X \mu(dx)h(x)=KC\sum_{\om\in\Om}(1+m_{t+\ep\om}(x_0))<\infty.  
\end{equation*}
So, again by the the dominated convergence theorem, (3) follows. $\Box$
