8
$\begingroup$

Suppose two probability density functions, $p$ and $q$, such that $\text{KL}(q||p) = \text{KL}(p||q) \neq 0$. Intuitively, does that tell us anything interesting about the nature of these densities?

$\endgroup$
1
  • $\begingroup$ If the divergence is finite, then the $q$ and $p$ must have the same support, right? $\endgroup$ – Bill Bradley Nov 20 '19 at 18:30
5
$\begingroup$

One example where this happens is when $P$ and $Q$ are "antipodal" distributions -- say, $p=(p_1,\ldots,p_n)$, $q=(q_1,\ldots,q_n)$, $q_i=1-p_i$, and $$P=\mathrm{Ber}(p_1)\times\mathrm{Ber}(p_2)\times\ldots \mathrm{Ber}(p_n)$$ and $Q$ is defined analogously. Then $KL(P||Q)=KL(Q||P)$. These play a role in optimal decision theory, see http://jmlr.org/beta/papers/v16/berend15a.html and https://projecteuclid.org/euclid.aos/1564797865 .

$\endgroup$
5
$\begingroup$

We're looking at the equation $$-\sum_{x\in\mathcal{X}} P(x) \log\left(\frac{Q(x)}{P(x)}\right) = -\sum_{x\in\mathcal{X}} Q(x) \log\left(\frac{P(x)}{Q(x)}\right). $$ In the Bernoulli case, $$-p \log\left(\frac{q}{p}\right) -(1-p) \log\left(\frac{1-q}{1-p}\right) = -q \log\left(\frac{p}{q}\right) -(1-q) \log\left(\frac{1-p}{1-q}\right) $$ The only solutions are random variables $X$ and $Y$ with $X=1-Y$ (mentioned by @Aryeh) and the trivial solution $X=Y$:

Wolfram Alpha

$\endgroup$
1
$\begingroup$

I doubt much can be said. One example is where $p$ is a translation of $q$, but there are many others. I will use the notation and results from my answer at https://stats.stackexchange.com/questions/188903/intuition-on-the-kullback-leibler-kl-divergence/189758#189758, where the KL divergence is interpreted in the context of statistical hypothesis testing.

Let (assuming $P$ and $Q$ are equivalent measures) $$ \DeclareMathOperator{\KL}{KL} \KL(P || Q) = \int \log \frac{dP}{dQ} \; dP $$ If $$ \KL(P || Q)=\KL(Q || P) $$ using the interpretation from the link above, $\KL(P || Q)$ is the expected value under the alternative for the log likelihood ratio for the testing problem $H_0 \colon X \sim Q; H_1 \colon X \sim P$, while $\KL(Q || P)$ is the expected value under the alternative for the log likelihood test for the testing problem $H_0 \colon X \sim P; H_1 \colon X \sim Q$. So in this sense, this two testing problems are equally difficult.

I can see no reason why this should tell us much about the structure of $P$ and $Q$. Maybe one can conclude that $P, Q$ must have similar tail weights (the intuition behind that comment is the normal/t example in the linked post.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.