It is well known that of all the joint distributions $p(x,y)$ with fixed marginals $p(x),p(y)$, the one with the highest entropy is: $$ p(x,y)=p(x)p(y). $$

Suppose instead that we have conditionals. Namely:

Which probability distribution $p(x,y,z)$, with fixed $p(x|y)$ and $p(x|z)$, has maximal entropy?

is there even an explicit formula?

(Note: one is tempted to take $p(x,y,z)$ such that $p(x|y,z)=p(x|y)p(x|z)$, but I think this is misleading, right?)

Feel free to edit question and tags appropriately.


(Crossposted from Cross Validated, where nobody could answer.)

  • 2
    $\begingroup$ Do $x$, $y$ and $z$ take their values from finite sets? Perhaps, you could also comment on why you start with three variables (instead of two), and why only have $p(x|y)$ and $p(x|z)$ as constraints and not, say, $p(x|y,z)$, $p(y|z,x)$ and $p(z|x,y)$ or any other choice. $\endgroup$
    – Algernon
    Aug 9, 2015 at 8:24
  • $\begingroup$ A closed form solution for the max ent distribution with given marginals (beyond 1-marginals) is fairly well-unknown. (By "well-unknown" I mean "well-known that it's not known." It might even be known that no such thing exists... From a quick google, see p. 10 here.) Using Lagrange multipliers you can always write it in an exponential form, but the terms appearing in that exponential formula don't have simple closed formulae... $\endgroup$ Dec 7, 2015 at 5:22

2 Answers 2


You can find a full formula in http://arxiv.org/pdf/1512.00752v3.pdf


This is a cute problem, but in most cases, you will be able to reach an arbitrarily high entropy.

Basically, for any $p(z)$ such that : $\int p(z)p(x|z)p(x|y) dxdydz $ is finite, you have a joint distribution that respects your condition.

Depending on your p(x|y) function, all p(z) could respect that condition (for example, if $\int p(x|y)dy \leq B$ for all $x$), in which case you can reach an abitrary entropy from p(z) alone

  • 1
    $\begingroup$ I suspect the OP assumes $x,y,z$ are chosen from a finite set. $\endgroup$
    – Algernon
    Aug 9, 2015 at 8:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.