What does the KL being symmetric tell us about the distributions?

Question

Suppose two probability density functions, $p$ and $q$, such that $\text{KL}(q||p) = \text{KL}(p||q) \neq 0$. Intuitively, does that tell us anything interesting about the nature of these densities?

Answer 1

One example where this happens is when $P$ and $Q$ are "antipodal" distributions -- say, $p=(p_1,\ldots,p_n)$, $q=(q_1,\ldots,q_n)$, $q_i=1-p_i$, and $$P=\mathrm{Ber}(p_1)\times\mathrm{Ber}(p_2)\times\ldots \mathrm{Ber}(p_n)$$ and $Q$ is defined analogously. Then $KL(P||Q)=KL(Q||P)$. These play a role in optimal decision theory, see http://jmlr.org/beta/papers/v16/berend15a.html and https://projecteuclid.org/euclid.aos/1564797865 .

Answer 2

We're looking at the equation $$-\sum_{x\in\mathcal{X}} P(x) \log\left(\frac{Q(x)}{P(x)}\right) = -\sum_{x\in\mathcal{X}} Q(x) \log\left(\frac{P(x)}{Q(x)}\right). $$ In the Bernoulli case, $$-p \log\left(\frac{q}{p}\right) -(1-p) \log\left(\frac{1-q}{1-p}\right) = -q \log\left(\frac{p}{q}\right) -(1-q) \log\left(\frac{1-p}{1-q}\right) $$ The only solutions are random variables $X$ and $Y$ with $X=1-Y$ (mentioned by @Aryeh) and the trivial solution $X=Y$:

Answer 3

I doubt much can be said. One example is where $p$ is a translation of $q$, but there are many others. I will use the notation and results from my answer at https://stats.stackexchange.com/questions/188903/intuition-on-the-kullback-leibler-kl-divergence/189758#189758, where the KL divergence is interpreted in the context of statistical hypothesis testing.

Let (assuming $P$ and $Q$ are equivalent measures) $$ \DeclareMathOperator{\KL}{KL} \KL(P || Q) = \int \log \frac{dP}{dQ} \; dP $$ If $$ \KL(P || Q)=\KL(Q || P) $$ using the interpretation from the link above, $\KL(P || Q)$ is the expected value under the alternative for the log likelihood ratio for the testing problem $H_0 \colon X \sim Q; H_1 \colon X \sim P$, while $\KL(Q || P)$ is the expected value under the alternative for the log likelihood test for the testing problem $H_0 \colon X \sim P; H_1 \colon X \sim Q$. So in this sense, this two testing problems are equally difficult.

I can see no reason why this should tell us much about the structure of $P$ and $Q$. Maybe one can conclude that $P, Q$ must have similar tail weights (the intuition behind that comment is the normal/t example in the linked post.)