2
$\begingroup$

Observation 0: Given a finite set $X$, the probability distribution on $X$ with highest entropy is the uniform one. This is well known.

Observation 1: Given two finite sets $X,Y$ and two probability distributions, $p$ on $X$ and $q$ on $Y$, there always exists a probability distribution on $X\times Y$ which has these given distributions as marginals, namely the one $p(x)\,q(y)$ whose projections are independent, and it is the one with highest entropy. (See here and here.)

Question: Given three finite sets $X,Y,Z$ and three probability distributions, $p$ on $X\times Y$, $q$ on $X\times Z$ and $r$ on $Y\times Z$:

  • how can I decide (in function of $p,q,r$) whether there is a probability distribution on $X\times Y\times Z$ which has these given distributions as pairwise marginals? (I mean, is there a simpler way to state this than by writing the obvious linear feasibility problem expressing it?)

  • and more importantly, assuming there is one, how can I compute (in function of $p,q,r$) the one with highest entropy? does this distribution (or this problem) at least have a name?

I am interested both in what can theoretically be said about this problem/distribution, and about computing it in practice. We can assume $|X|=|Y|=|Z|=2$ if this is of help.

(A similar question was asked here, but it's not quite the same setup, and not as symmetric.)

Improve this question
$\endgroup$
5
  • $\begingroup$ Just to clarify: with respect to which reference measure are you computing your entropy? $\endgroup$
    – leo monsaingeon
    Commented Sep 1, 2023 at 22:18
  • $\begingroup$ Also, I presume you enforce the obvious compatibility between the two-marginales $p,q,r$, otherwise there might not even exist a joint distribution on $X\times Y\times Z$ (in your notations one must have for example $\pi^x\# p =\pi^x\#q$ and so on, by permutation) $\endgroup$
    – leo monsaingeon
    Commented Sep 1, 2023 at 22:22
  • $\begingroup$ One last comment: if one of your three 2-marginals is induced by a map (say for example $p(dx,dy)=\mu(dx)\otimes \delta_{T(x)}(dy)$ for some map $T:X\to Y$) then there exists only one posisble three-distribution, this is given for example in lemma 5.3.2 in the book Gradient flows in metric spaces [..].by L. Ambrosio, N. Gigli, and G. Savaré. So in that case the maximization is not really interesting. But I guess for probabilists the case of transport-induced couplings is not particulartly relevant, is it? $\endgroup$
    – leo monsaingeon
    Commented Sep 1, 2023 at 22:25
  • 1
    $\begingroup$ Note that in the case |X|=|Y|=|Z|=2, constraining the 2-marginals is the same as constraining the moments of degree <=2, so the log of the maximum entropy distribution is a degree-2 polynomial. Solving in practice is then an exercise in root-finding (solving for the coefficients of the polynomial that give the desired marginals/moments). $\endgroup$
    – Yoav Kallus
    Commented Sep 2, 2023 at 1:31
  • $\begingroup$ @leomonsaingeon Sorry, the entropy is computed wrt the uniform distribution on every finite set involved. And yes, there is a compatibility condition on $p,q,r$: the first part of my question is whether there is a simple way to express it (but I guess this isn't really interesting unless it also helps answer the second part). $\endgroup$
    – Gro-Tsen
    Commented Sep 2, 2023 at 9:23

1 Answer 1

Reset to default
1
$\begingroup$

This is only about $|X|=|Y|=|Z|=2$ case: let $X=Y=Z=\{0,1\}$. And $f$ be a "seed" joint distribution on $X\times Y\times Z$. Then if another distribution has same pairwise marginals, there difference is one dimensional: $\mu$ on $(0,0,0),(1,1,0),(1,0,1),(0,1,1)$ and $-\mu$ on compliment.

Then maximizing entropy is equivalent to $(f(0,0,0)+\mu)(f(1,1,0)+\mu)(f(1,0,1)+\mu)(f(0,1,1)+\mu)=(f(1,0,0)-\mu)(f(0,1,0)-\mu)(f(0,0,1)-\mu)(f(1,1,1)-\mu)$. Which seems no simple expression of roots are available.

Some updates on general case: for every 2-subset of $X,Y$ and $Z$, there is such pertubations and if $f$ is the distribution maximizing entropy, it should satisfy $f(x_0,y_0,z_0)f(x_1,y_1,z_0)f(x_1,y_0,z_1)f(x_0,y_1,z_1)=f(x_1,y_0,z_0)f(x_0,y_1,z_0)f(x_0,y_0,z_1)f(x_1,y_1,z_1)$. Which indicates $f(x,y,z)=\exp(u(x,y)+v(y,z)+w(x,z))$ for some $u,v,w$. It's not hard to show it's also a sufficient condition.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .