8
$\begingroup$

This is an analog of an older question:

What characterizations of relative information are known?

With the modification that I’m interested in the case when the distribution is over something that’s not a finite set. For example, for compactly supported distributions over an interval equipped with some measure. The definition of the KL divergence in this case is found as the third equation in the defintions section in the relevant wikipedia entry.

I would like to know whether there’s an axiomatic characterization of this, generalizing the characterizations in the discrete case.

My limited intuition (as a non-information-theorist) is that this could be tricky, for I’m reminded that there’s a nice characterization of ordinary entropy of discrete distributions due to Fadeev, which lacks an obvious generalization to the differential/continuous entropy. There’s a relevant discussion of this issue in another older post.

$\endgroup$
3
  • $\begingroup$ Possible duplicate of Is there an axiomatic characterization of the entropy of a continuous random variable? $\endgroup$
    – R W
    Sep 23, 2018 at 18:45
  • 1
    $\begingroup$ @RW relative entropy/KL divergence is a different object from the one discussed in the linked post (ordinary entropy) , with different properties. $\endgroup$
    – zzz
    Sep 23, 2018 at 18:47
  • $\begingroup$ I'm the author of the other question, and I agree that ordinary entropy and relative entropy for continuous distributions are different enough objects that this deserves to be a separate question. Ordinary entropy, unlike relative entropy, varies when you change variables, and it has no positive definiteness property. So I would naively expect relative to entropy to have a different (and better) axiomatic characterization. $\endgroup$ Dec 2, 2020 at 12:31

1 Answer 1

1
$\begingroup$

Skilling wrote a paper with an axiomatic approach to the relative entropy [1] as a tool for inductive reasoning. And furthermore, there is another paper by Caticha and Preuss [2], which nicely summarises Skilling's approach.

[1] John Skilling (1988). The axioms of maximum entropy.

[2] Ariel Caticha, Roland Preuss (2003). Maximum Entropy and Bayesian Data Analysis: Entropic Priors.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.