# What does the KL being symmetric tell us about the distributions?

Suppose two probability density functions, $$p$$ and $$q$$, such that $$\text{KL}(q||p) = \text{KL}(p||q) \neq 0$$. Intuitively, does that tell us anything interesting about the nature of these densities?

• If the divergence is finite, then the $q$ and $p$ must have the same support, right? – Bill Bradley Nov 20 '19 at 18:30

One example where this happens is when $$P$$ and $$Q$$ are "antipodal" distributions -- say, $$p=(p_1,\ldots,p_n)$$, $$q=(q_1,\ldots,q_n)$$, $$q_i=1-p_i$$, and $$P=\mathrm{Ber}(p_1)\times\mathrm{Ber}(p_2)\times\ldots \mathrm{Ber}(p_n)$$ and $$Q$$ is defined analogously. Then $$KL(P||Q)=KL(Q||P)$$. These play a role in optimal decision theory, see http://jmlr.org/beta/papers/v16/berend15a.html and https://projecteuclid.org/euclid.aos/1564797865 .

We're looking at the equation $$-\sum_{x\in\mathcal{X}} P(x) \log\left(\frac{Q(x)}{P(x)}\right) = -\sum_{x\in\mathcal{X}} Q(x) \log\left(\frac{P(x)}{Q(x)}\right).$$ In the Bernoulli case, $$-p \log\left(\frac{q}{p}\right) -(1-p) \log\left(\frac{1-q}{1-p}\right) = -q \log\left(\frac{p}{q}\right) -(1-q) \log\left(\frac{1-p}{1-q}\right)$$ The only solutions are random variables $$X$$ and $$Y$$ with $$X=1-Y$$ (mentioned by @Aryeh) and the trivial solution $$X=Y$$:

I doubt much can be said. One example is where $$p$$ is a translation of $$q$$, but there are many others. I will use the notation and results from my answer at https://stats.stackexchange.com/questions/188903/intuition-on-the-kullback-leibler-kl-divergence/189758#189758, where the KL divergence is interpreted in the context of statistical hypothesis testing.

Let (assuming $$P$$ and $$Q$$ are equivalent measures) $$\DeclareMathOperator{\KL}{KL} \KL(P || Q) = \int \log \frac{dP}{dQ} \; dP$$ If $$\KL(P || Q)=\KL(Q || P)$$ using the interpretation from the link above, $$\KL(P || Q)$$ is the expected value under the alternative for the log likelihood ratio for the testing problem $$H_0 \colon X \sim Q; H_1 \colon X \sim P$$, while $$\KL(Q || P)$$ is the expected value under the alternative for the log likelihood test for the testing problem $$H_0 \colon X \sim P; H_1 \colon X \sim Q$$. So in this sense, this two testing problems are equally difficult.

I can see no reason why this should tell us much about the structure of $$P$$ and $$Q$$. Maybe one can conclude that $$P, Q$$ must have similar tail weights (the intuition behind that comment is the normal/t example in the linked post.)