# Minimizing KL divergence: the asymmetry, when will the solution be the same?

The KL divergence between two distribution $p$ and $q$ is defined as $$D( q \| p)\int q(x)\log \frac{q(x)}{p(x)} dx$$ and is known to be asymmetry: $D(q\|p)\neq D(p\|q)$.

If we fix $p$ and try to find a distribution $q$ among a class $E$ that minimize the KL distance, it is also known that minimizing $D( q \| p)$ will be different from minimizing $D( p \| q)$, e.g., https://benmoran.wordpress.com/2012/07/14/kullback-leibler-divergence-asymmetry/.

It is not clear which one to be optimized for a better approximation, although in many application we minimize $D(q\|p)$.

My question is that, when will the solution be the same

$$\underset{q\in E}{\operatorname{argmin}} D(q\|p) = \underset{q\in E}{\operatorname{argmin}} D(p\|q)?$$

For instance, if we take $E$ as the class of all Gaussian distribution, is there a condition on $p$ so that minimizing these two will lead to the same minimizer?

• I'm a little confused: isn't $D(q || p)$ minimized at $q = p$ since $p \log \frac{p}{p} = 0$? Isn't it more normal to maximize KL-divergence? Apr 28 '17 at 12:36
• Well, OP minimizes over a constraint set $E$, i.e. seeking the "closest distribution to $p$ that comes from $E$". Makes sense to me.
– Dirk
Apr 28 '17 at 13:57

Formulate the optimization problems with constraints as $$\mathrm{argmin}_{F(q)=0} D(q || p),\qquad \mathrm{argmin}_{F(q)=0} D(p||q)$$ and form the respective Lagrange functionals. Using that the derivatives of $D$ w.r.t. to the first and second components are, respectively, $$\nabla_1 D(q||p) = \log(\tfrac{q}{p})+1\quad\text{and}\quad \nabla_2 D(p||q) = \tfrac{q}{p}$$ you see that necessary conditions for optima $q^*$ and $q^{**}$, respectively, are $$\log(\tfrac{q^*}{p})+1 + \nabla F(q^*)\lambda = 0\quad\text{and}\quad \tfrac{q^{**}}{p} + \nabla F(q^{**})\lambda = 0.$$ I would not expect that $q^*$ and $q^{**}$ are equal for any non-trivial constraint…
On the positive side, $\nabla_1 D(q||p)$ and $\nabla_2 D(q||p)$ agree up to first order at $p=q$, i.e. $$\nabla_1 D(q||p) = \nabla_2 D(q||p) + \mathcal{O}(\tfrac{q}{p})$$.
• Yes, it's simply $(D(p||q)+D(q||p))/2.$ Apr 30 '17 at 7:05