I am interesting in axiomatic justifications for concepts in information theory. I have found many axiomatizations for Shannon's entropy and for the Kullback-Leibler divergence, as well as their variations (for some examples, see this survey or these books: 1, 2). However, I have not found any results directly for the mutual information.

Are there any results that show that some intuitive properties uniquely characterize the mutual information of a joint distribution of two random variables?

  • $\begingroup$ Can I rephrase this as : Given a random sequencegenerated by a random variable X and another by a random variable Y, is their an algorithm to find the mutual information of X and Y. You want an axiomatic justification of info theory argument? $\endgroup$
    – ARi
    Jun 19, 2013 at 5:44
  • $\begingroup$ I think I have in mind something different. All I'm saying is: look at the mutual information as a function from pairs of random variables to real numbers. I'm looking for a theorem of the form: a function (with this domain and codomain) satisfies properties 1 to n if and only if it's the mutual information of the two random variables. $\endgroup$ Jun 19, 2013 at 19:02
  • 1
    $\begingroup$ I'm not sure if I've seen such a characterization before, but since mutual information is a KL divergence, it might be possible to add axioms to those for the KL divergence... $\endgroup$ Aug 23, 2013 at 20:38

1 Answer 1


My paper https://arxiv.org/pdf/2108.12647.pdf provides an answer to this question. In particular, here is a quote from the introduction:

Our main result is Theorem~6.6, where we prove that the mutual information $\mathbb{I}(\mathcal{X},\mathcal{Y})$ of an ordered pair of random variables is the unique function (up to an arbitrary multiplicative factor) on pairs of random variables satisfying the following axioms: enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.