Convex support of an exponential family and its mean parameter space $\mathcal{M}$

Question

This question comes up in studying mean parametrization of exponential families of distributions. (See Brown's 1986 book on the subject.)

Let $\nu$ be a (Borel) measure on $\mathbb R^d$. Let $p(\cdot)$ denote a generic probability density w.r.t $\nu$, and let \begin{align} \mathcal M &:= \Big\{ \int x p(x) d\nu(x):\; \int p(x) d\nu(x) = 1\Big\} \\&:= \Big\{ \int x d P:\; P(\mathbb R^d) = 1, \quad P \ll \nu\Big\} \end{align} be the set of realizable means by probability measures absolutely continuous w.r.t. $\nu$.

Now, let $\text{supp}(\nu)$ be the support of $\nu$, the smallest closed set whose complement has $\nu$-measure zero. Let $$ \mathcal K = \overline{\text{conv}}(\text{supp}(\nu)) $$ where $\overline{\text{conv}}$ denotes the closed convex hull of a set. Brown calls $\mathcal K$ the convex support of $\nu$. (Is this a standard terminology?)

It seems that $\mathcal M$ and $\mathcal K$ are close, say $\mathcal M \subset \mathcal K \subset \overline{\mathcal M}$ (?) and the two could only be different over the boundaries of the two set. Any references shedding light on the relationship, esp. what happens at the boundary of these sets, is appreciated. (There seems to be a vague connection to Choquet theory?)

An example: Let $\nu$ be the push-forward of the 1-D Lebesgue measure by the map $x \mapsto (x,x^2)$, then the set $\mathcal M = \{(\mu_1,\mu_2) :\; \mu_2 > \mu_1^2 \}$ which is an open set. To see this, we note that $(\mu_1,\mu_2) \in \mathcal M$ iff $(\mu_1,\mu_2) = E (X,X^2)$ where $X$ is a random variable whose distribution is absolutely continuous w.r.t. the Lebesgue measure. We always have $\mu_2 \ge \mu_1^2$ (by Jensen inequality) and anything with $\mu_2 > \mu_1^2$ can be realized by a non-degenerate Gaussian distribution with mean $\mu_1$ and variance $\mu_2 - \mu_1^2$. Anything on the boundary $\mu_2 = \mu_1^2$ corresponds to a point mass (variance = 0), hence cannot be realized by a distribution absolutely continuous w.r.t. the Lebesgue measure.

In fact, the interesting result is that anything in interior of $\mathcal K$ (or shall we say interior of $\mathcal M$) can be realized by the corresponding exponential family. This is Theorem 3.6 in Brown. The above example is a special case of this, where the Gaussian is the corresponding exponential family distribution.

Answer 1

I have to admit that this is the first time I have heard about the term "Choquet Theory" although I heard his name in my topology class earlier. After I read the wikipedia page (Choquet theory) I knew that is the general case of the "Carathéodory Theorem"(Carathéodory's theorem) I usually refer to, thanks for this nice lesson! And yes it does relate with what you asked in OP.

But let me answer your question in a few parts.

$\blacksquare$1.What role does mean parameter space play in probability/statistics?

In the setting of a regression analysis using a non-degenerated (Wronskian$\neq$0) basis $\{u_0,\cdots u_n\}$. The mean parameter space is realized by letting $pd\nu$ traversing on $M(X)$, all possible probability measures defined on $X$(a compact subset of $\mathbb{R}^d$ if you want Euclidean structure built into your sample space.) $$\mathcal{M}_{n+1}=\{(c_0,\cdots c_n); c_i=\int u_i(x)p(x) d\nu(x),\forall p\in M(X)\} $$ is the smallest affine convex cone containing the curves determined by coordinate $(u_0,\cdots u_n)$ if $X$ is compact, say $X=[a_i,b_i]^d$.

More specifically, we consider the canonical case where $u_i:=x^i,i=0,1,\cdots n$ are the moments of the random variable $X$ corresponding to the p.d.f. $p(x)$ w.r.t. $\nu$, then we can prove that $\mathcal{M}$ is a closed convex cone that contains the (rectifiable) curve $(1,x,\cdots x^n)\subset \mathbb{A}^{(n+1)}$.

For a sketch of the proof, $\sum_i \lambda_i x^i \in \mathcal{M}_{n+1},\lambda_i\geq 0$ is clear since we can always choose mixture of $p(x)$ to realize particular coordinate. Conversely, if there exists $c=\sum_i \lambda_i x^i \notin \mathcal{M}_{n+1}$, then there existing a separating hyperplane (linear functional) separating the convex conic hull of the curve and $c$, we can verify that this separating hyperplane is not possible at the apex of the convex cone and thus lead to contradiction. Details are in [Rockafellar] which makes use of Carathéodory's theorem to argue that the convex conic hull can be defined as the collection of all curves in form of $\sum_i \lambda_i x^i \in \mathcal{M}_{n+1},\lambda_i\geq 0$.

Therefore when $n=1$ and the mean parameter space becomes $\mathcal{M}_{2}$ as you stated in the OP. It is simply a 2-dim convex conic hull of the curve generated by $(1,x)$ as $x$ varies in $X$.

$\blacksquare$2.Is this a standard terminology?

I believe so. See also [Rockafellar].

$\blacksquare$3.Closedness of $\mathcal{M}$

This is guaranteed by the Helly's selection theorem applied on the $pd\nu$. Thus it is meaningless to inquire $\mathcal M \subset \mathcal K \subset \overline{\mathcal M}$ since $\mathcal M =\overline{\mathcal M}$. Instead you can ask $\mathcal M ? \supset \mathcal K$. I think the geometric aspect can be found in [Caratheodory].

By the duality $\mathcal{P}_{n+1}=\mathcal{M}_{n+1}^{+}$ where $\mathcal{M}^{+}:=\{\boldsymbol{a};\boldsymbol{a'c}\geq 0, \forall \boldsymbol{c}\in \mathcal{M} \}$ between nonnegative combinations $$\mathcal{P}_{n+1}:=\{(a_0,\cdots a_n);\sum_j a_j u_j(x)\geq 0,\forall x\in X\}$$ we know that the $\mathcal{P}_{n+1}$ is the dual cone of the moment space and hence also closed. Therefore the boundary of $\mathcal{K}$ and $\mathcal{M}$ will only intersect when $\sum_j a_j u_j(x)= 0$ which is $a_0+a_1\cdot x= 0$ in your $\mathcal{M}_{2}$ case.

Another answer from perspective of statistcians is https://stats.stackexchange.com/questions/266691/is-the-expectation-of-the-sufficient-statistics-sx-transverse-the-whole-spac.

Reference

[Rockafellar]Rockafellar, Ralph Tyrell. Convex analysis. Princeton university press, 2015.

[Caratheodory]Carathéodory, Constantin. "Über den Variabilitätsbereich der Koeffizienten von Potenzreihen, die gegebene Werte nicht annehmen." Mathematische Annalen 64.1 (1907): 95-115.

[Brown]Brown, Lawrence D. "Fundamentals of statistical exponential families with applications in statistical decision theory." Lecture Notes-monograph series 9 (1986): i-279.