# Convex support of an exponential family and its mean parameter space $\mathcal{M}$

This question comes up in studying mean parametrization of exponential families of distributions. (See Brown's 1986 book on the subject.)

Let $\nu$ be a (Borel) measure on $\mathbb R^d$. Let $p(\cdot)$ denote a generic probability density w.r.t $\nu$, and let \begin{align} \mathcal M &:= \Big\{ \int x p(x) d\nu(x):\; \int p(x) d\nu(x) = 1\Big\} \\&:= \Big\{ \int x d P:\; P(\mathbb R^d) = 1, \quad P \ll \nu\Big\} \end{align} be the set of realizable means by probability measures absolutely continuous w.r.t. $\nu$.

Now, let $\text{supp}(\nu)$ be the support of $\nu$, the smallest closed set whose complement has $\nu$-measure zero. Let $$\mathcal K = \overline{\text{conv}}(\text{supp}(\nu))$$ where $\overline{\text{conv}}$ denotes the closed convex hull of a set. Brown calls $\mathcal K$ the convex support of $\nu$. (Is this a standard terminology?)

It seems that $\mathcal M$ and $\mathcal K$ are close, say $\mathcal M \subset \mathcal K \subset \overline{\mathcal M}$ (?) and the two could only be different over the boundaries of the two set. Any references shedding light on the relationship, esp. what happens at the boundary of these sets, is appreciated. (There seems to be a vague connection to Choquet theory?)

An example: Let $\nu$ be the push-forward of the 1-D Lebesgue measure by the map $x \mapsto (x,x^2)$, then the set $\mathcal M = \{(\mu_1,\mu_2) :\; \mu_2 > \mu_1^2 \}$ which is an open set. To see this, we note that $(\mu_1,\mu_2) \in \mathcal M$ iff $(\mu_1,\mu_2) = E (X,X^2)$ where $X$ is a random variable whose distribution is absolutely continuous w.r.t. the Lebesgue measure. We always have $\mu_2 \ge \mu_1^2$ (by Jensen inequality) and anything with $\mu_2 > \mu_1^2$ can be realized by a non-degenerate Gaussian distribution with mean $\mu_1$ and variance $\mu_2 - \mu_1^2$. Anything on the boundary $\mu_2 = \mu_1^2$ corresponds to a point mass (variance = 0), hence cannot be realized by a distribution absolutely continuous w.r.t. the Lebesgue measure.

In fact, the interesting result is that anything in interior of $\mathcal K$ (or shall we say interior of $\mathcal M$) can be realized by the corresponding exponential family. This is Theorem 3.6 in Brown. The above example is a special case of this, where the Gaussian is the corresponding exponential family distribution.

• You don't want a proof of the second inequality ? I don't know what Brown was interested in, but the question of when the risk free rate is in M is an important question in finance and you might browse the introductory chapters of Duffie's book Dynamic Asset Pricing & see what it suggests.
– user83457
Feb 20, 2017 at 13:46
• @passerby51 I edited the answer a bit and hope that solve your question. Feb 21, 2017 at 11:45
• @michael, that is interesting. It would be great if you could explain the connection with finance in an answer and yes I am interested in any proof of the second inequality. Brown was perhaps motivated by statistical applications, though he does not explicitly introduce $\mathcal M$, that is perhaps a more recent perspective. He works with the interior of $\mathcal K$. Feb 21, 2017 at 17:02
• @Henry.L, thanks, please see my added example. Feb 21, 2017 at 17:03
• @passerby51 When you relax the compactness assumption on the $X$, you can say $\mathcal{M}\subset\mathcal{\bar{M}}$ trivially, and I think I have made it clear that duality relation implies $supp p(x)\subset\mathcal{K}$, and $\mathcal{M}\subset supp p(x)$. The second inequality is almost trivial since $suppp(x)\subset\mathcal{\bar{M}}$ is clear by Hamburger moment problem. I have corrected and emphasized the requirement of compactness. Hope helps. Feb 21, 2017 at 20:11

I have to admit that this is the first time I have heard about the term "Choquet Theory" although I heard his name in my topology class earlier. After I read the wikipedia page (Choquet theory) I knew that is the general case of the "Carathéodory Theorem"(Carathéodory's theorem) I usually refer to, thanks for this nice lesson! And yes it does relate with what you asked in OP.

$\blacksquare$1.What role does mean parameter space play in probability/statistics?

In the setting of a regression analysis using a non-degenerated (Wronskian$\neq$0) basis $\{u_0,\cdots u_n\}$. The mean parameter space is realized by letting $pd\nu$ traversing on $M(X)$, all possible probability measures defined on $X$(a compact subset of $\mathbb{R}^d$ if you want Euclidean structure built into your sample space.) $$\mathcal{M}_{n+1}=\{(c_0,\cdots c_n); c_i=\int u_i(x)p(x) d\nu(x),\forall p\in M(X)\}$$ is the smallest affine convex cone containing the curves determined by coordinate $(u_0,\cdots u_n)$ if $X$ is compact, say $X=[a_i,b_i]^d$.

More specifically, we consider the canonical case where $u_i:=x^i,i=0,1,\cdots n$ are the moments of the random variable $X$ corresponding to the p.d.f. $p(x)$ w.r.t. $\nu$, then we can prove that $\mathcal{M}$ is a closed convex cone that contains the (rectifiable) curve $(1,x,\cdots x^n)\subset \mathbb{A}^{(n+1)}$.

For a sketch of the proof, $\sum_i \lambda_i x^i \in \mathcal{M}_{n+1},\lambda_i\geq 0$ is clear since we can always choose mixture of $p(x)$ to realize particular coordinate. Conversely, if there exists $c=\sum_i \lambda_i x^i \notin \mathcal{M}_{n+1}$, then there existing a separating hyperplane (linear functional) separating the convex conic hull of the curve and $c$, we can verify that this separating hyperplane is not possible at the apex of the convex cone and thus lead to contradiction. Details are in [Rockafellar] which makes use of Carathéodory's theorem to argue that the convex conic hull can be defined as the collection of all curves in form of $\sum_i \lambda_i x^i \in \mathcal{M}_{n+1},\lambda_i\geq 0$.

Therefore when $n=1$ and the mean parameter space becomes $\mathcal{M}_{2}$ as you stated in the OP. It is simply a 2-dim convex conic hull of the curve generated by $(1,x)$ as $x$ varies in $X$.

$\blacksquare$2.Is this a standard terminology?

$\blacksquare$3.Closedness of $\mathcal{M}$

This is guaranteed by the Helly's selection theorem applied on the $pd\nu$. Thus it is meaningless to inquire $\mathcal M \subset \mathcal K \subset \overline{\mathcal M}$ since $\mathcal M =\overline{\mathcal M}$. Instead you can ask $\mathcal M ? \supset \mathcal K$. I think the geometric aspect can be found in [Caratheodory].

By the duality $\mathcal{P}_{n+1}=\mathcal{M}_{n+1}^{+}$ where $\mathcal{M}^{+}:=\{\boldsymbol{a};\boldsymbol{a'c}\geq 0, \forall \boldsymbol{c}\in \mathcal{M} \}$ between nonnegative combinations $$\mathcal{P}_{n+1}:=\{(a_0,\cdots a_n);\sum_j a_j u_j(x)\geq 0,\forall x\in X\}$$ we know that the $\mathcal{P}_{n+1}$ is the dual cone of the moment space and hence also closed. Therefore the boundary of $\mathcal{K}$ and $\mathcal{M}$ will only intersect when $\sum_j a_j u_j(x)= 0$ which is $a_0+a_1\cdot x= 0$ in your $\mathcal{M}_{2}$ case.

Another answer from perspective of statistcians is https://stats.stackexchange.com/questions/266691/is-the-expectation-of-the-sufficient-statistics-sx-transverse-the-whole-spac.

Reference

[Rockafellar]Rockafellar, Ralph Tyrell. Convex analysis. Princeton university press, 2015.

[Caratheodory]Carathéodory, Constantin. "Über den Variabilitätsbereich der Koeffizienten von Potenzreihen, die gegebene Werte nicht annehmen." Mathematische Annalen 64.1 (1907): 95-115.

[Brown]Brown, Lawrence D. "Fundamentals of statistical exponential families with applications in statistical decision theory." Lecture Notes-monograph series 9 (1986): i-279.

• thanks for the detailed response. Not sure if I follow everything. I have added an example to my post for which I believe $\mathcal M$ is open. Am I missing something? Feb 21, 2017 at 16:58
• @passerby51 A curved exponential family is still an exponential family, what you raised in the example is just $N(\mu,\mu)\in\partial\mathcal{M}$, a variance zero degenerate distribution is the apex of the $\mathcal{M}$. Feb 21, 2017 at 19:53
• @passerby51 And by saying that $\mathcal{M}$ is closed by Helly's selection theorem because any limit of BV functions will yet still converge to a BV function. Note that I emphasize $X$ is compact in order to make my argument hold, your example is on $\mathbb{R}^d$,which is not compact. My comment in the bracket is to say that you can choose $\mathbb{R}^{d}$ as the underlying imbedded space, which means you want to inherit Euclidean structure. I do not mean you can choose $X=\mathbb{R}^d$ and still hope the interpretation to be valid. I have make it clear now. Feb 21, 2017 at 19:59
• @passerby51 A more detail explanation is available now on stats.stackexchange.com/questions/266691/…, hope helps! Mar 23, 2017 at 23:40