This is an analog of an older question:
What characterizations of relative information are known?
With the modification that I’m interested in the case when the distribution is over something that’s not a finite set. For example, for compactly supported distributions over an interval equipped with some measure. The definition of the KL divergence in this case is found as the third equation in the defintions section in the relevant wikipedia entry.
I would like to know whether there’s an axiomatic characterization of this, generalizing the characterizations in the discrete case.
My limited intuition (as a non-information-theorist) is that this could be tricky, for I’m reminded that there’s a nice characterization of ordinary entropy of discrete distributions due to Fadeev, which lacks an obvious generalization to the differential/continuous entropy. There’s a relevant discussion of this issue in another older post.