# Maximal entropy distribution on three variables knowing its marginals on any two

Observation 0: Given a finite set $$X$$, the probability distribution on $$X$$ with highest entropy is the uniform one. This is well known.

Observation 1: Given two finite sets $$X,Y$$ and two probability distributions, $$p$$ on $$X$$ and $$q$$ on $$Y$$, there always exists a probability distribution on $$X\times Y$$ which has these given distributions as marginals, namely the one $$p(x)\,q(y)$$ whose projections are independent, and it is the one with highest entropy. (See here and here.)

Question: Given three finite sets $$X,Y,Z$$ and three probability distributions, $$p$$ on $$X\times Y$$, $$q$$ on $$X\times Z$$ and $$r$$ on $$Y\times Z$$:

• how can I decide (in function of $$p,q,r$$) whether there is a probability distribution on $$X\times Y\times Z$$ which has these given distributions as pairwise marginals? (I mean, is there a simpler way to state this than by writing the obvious linear feasibility problem expressing it?)

• and more importantly, assuming there is one, how can I compute (in function of $$p,q,r$$) the one with highest entropy? does this distribution (or this problem) at least have a name?

I am interested both in what can theoretically be said about this problem/distribution, and about computing it in practice. We can assume $$|X|=|Y|=|Z|=2$$ if this is of help.

(A similar question was asked here, but it's not quite the same setup, and not as symmetric.)

• Just to clarify: with respect to which reference measure are you computing your entropy? Commented Sep 1, 2023 at 22:18
• Also, I presume you enforce the obvious compatibility between the two-marginales $p,q,r$, otherwise there might not even exist a joint distribution on $X\times Y\times Z$ (in your notations one must have for example $\pi^x\# p =\pi^x\#q$ and so on, by permutation) Commented Sep 1, 2023 at 22:22
• One last comment: if one of your three 2-marginals is induced by a map (say for example $p(dx,dy)=\mu(dx)\otimes \delta_{T(x)}(dy)$ for some map $T:X\to Y$) then there exists only one posisble three-distribution, this is given for example in lemma 5.3.2 in the book Gradient flows in metric spaces [..].by L. Ambrosio, N. Gigli, and G. Savaré. So in that case the maximization is not really interesting. But I guess for probabilists the case of transport-induced couplings is not particulartly relevant, is it? Commented Sep 1, 2023 at 22:25
• Note that in the case |X|=|Y|=|Z|=2, constraining the 2-marginals is the same as constraining the moments of degree <=2, so the log of the maximum entropy distribution is a degree-2 polynomial. Solving in practice is then an exercise in root-finding (solving for the coefficients of the polynomial that give the desired marginals/moments). Commented Sep 2, 2023 at 1:31
• @leomonsaingeon Sorry, the entropy is computed wrt the uniform distribution on every finite set involved. And yes, there is a compatibility condition on $p,q,r$: the first part of my question is whether there is a simple way to express it (but I guess this isn't really interesting unless it also helps answer the second part). Commented Sep 2, 2023 at 9:23

This is only about $$|X|=|Y|=|Z|=2$$ case: let $$X=Y=Z=\{0,1\}$$. And $$f$$ be a "seed" joint distribution on $$X\times Y\times Z$$. Then if another distribution has same pairwise marginals, there difference is one dimensional: $$\mu$$ on $$(0,0,0),(1,1,0),(1,0,1),(0,1,1)$$ and $$-\mu$$ on compliment.
Then maximizing entropy is equivalent to $$(f(0,0,0)+\mu)(f(1,1,0)+\mu)(f(1,0,1)+\mu)(f(0,1,1)+\mu)=(f(1,0,0)-\mu)(f(0,1,0)-\mu)(f(0,0,1)-\mu)(f(1,1,1)-\mu)$$. Which seems no simple expression of roots are available.
Some updates on general case: for every 2-subset of $$X,Y$$ and $$Z$$, there is such pertubations and if $$f$$ is the distribution maximizing entropy, it should satisfy $$f(x_0,y_0,z_0)f(x_1,y_1,z_0)f(x_1,y_0,z_1)f(x_0,y_1,z_1)=f(x_1,y_0,z_0)f(x_0,y_1,z_0)f(x_0,y_0,z_1)f(x_1,y_1,z_1)$$. Which indicates $$f(x,y,z)=\exp(u(x,y)+v(y,z)+w(x,z))$$ for some $$u,v,w$$. It's not hard to show it's also a sufficient condition.