The KL divergence between two distribution $p$ and $q$ is defined as $$ D( q \| p)\int q(x)\log \frac{q(x)}{p(x)} dx $$ and is known to be asymmetry: $D(q\|p)\neq D(p\|q)$.

If we fix $p$ and try to find a distribution $q$ among a class $E$ that minimize the KL distance, it is also known that minimizing $D( q \| p)$ will be different from minimizing $D( p \| q)$, e.g., https://benmoran.wordpress.com/2012/07/14/kullback-leibler-divergence-asymmetry/.

It is not clear which one to be optimized for a better approximation, although in many application we minimize $D(q\|p)$.

My question is that, when will the solution be the same

$$ \underset{q\in E}{\operatorname{argmin}} D(q\|p) = \underset{q\in E}{\operatorname{argmin}} D(p\|q)? $$

For instance, if we take $E$ as the class of all Gaussian distribution, is there a condition on $p$ so that minimizing these two will lead to the same minimizer?

  • 2
    $\begingroup$ I'm a little confused: isn't $D(q || p)$ minimized at $q = p$ since $p \log \frac{p}{p} = 0$? Isn't it more normal to maximize KL-divergence? $\endgroup$ Apr 28 '17 at 12:36
  • 1
    $\begingroup$ Well, OP minimizes over a constraint set $E$, i.e. seeking the "closest distribution to $p$ that comes from $E$". Makes sense to me. $\endgroup$
    – Dirk
    Apr 28 '17 at 13:57

I don't have a definite answer, but here is something to continue with:

Formulate the optimization problems with constraints as $$ \mathrm{argmin}_{F(q)=0} D(q || p),\qquad \mathrm{argmin}_{F(q)=0} D(p||q) $$ and form the respective Lagrange functionals. Using that the derivatives of $D$ w.r.t. to the first and second components are, respectively, $$ \nabla_1 D(q||p) = \log(\tfrac{q}{p})+1\quad\text{and}\quad \nabla_2 D(p||q) = \tfrac{q}{p} $$ you see that necessary conditions for optima $q^*$ and $q^{**}$, respectively, are $$ \log(\tfrac{q^*}{p})+1 + \nabla F(q^*)\lambda = 0\quad\text{and}\quad \tfrac{q^{**}}{p} + \nabla F(q^{**})\lambda = 0. $$ I would not expect that $q^*$ and $q^{**}$ are equal for any non-trivial constraint…

On the positive side, $\nabla_1 D(q||p)$ and $\nabla_2 D(q||p)$ agree up to first order at $p=q$, i.e. $$\nabla_1 D(q||p) = \nabla_2 D(q||p) + \mathcal{O}(\tfrac{q}{p})$$.

  • 1
    $\begingroup$ I think there is a symmetrized version of KL divergence if the OP wants symmetry so much... But the motivation of doing so is not very clear as he described. $\endgroup$
    – Henry.L
    Apr 28 '17 at 12:38
  • $\begingroup$ Yes, it's simply $(D(p||q)+D(q||p))/2.$ $\endgroup$
    – kodlu
    Apr 30 '17 at 7:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.