# Derive KL divergence from Bregman divergence

I know that KL divergence is a form of Bregman divergence for multinomial distributions. Is there some derivation of KL divergence from functional Bregman divergence or some generalized Bregman divergence for general continuous distributions?

Start from the definition of the Bregman divergence, $$D_F(P,Q)=F(P)-F(Q)-\langle F'(Q), P-Q\rangle.$$ Substitute $$F(p)=\int dx\,p(x)\log p(x)$$ and work it out, $$D_F(P,Q)=\int dx\,p(x)\log p(x)-\int dx\,q(x)\log q(x)-\langle I+\log Q,P-Q\rangle$$ $$\qquad=\int dx\,p(x)\log p(x)-\int dx\,q(x)\log q(x)-\int dx\,\bigl(1+\log q(x)\bigr)\bigl(p(x)-q(x)\bigr)$$ $$\qquad=\int dx\,p(x)\log p(x)-\int dx\,p(x)\log q(x),$$ which is the definition of the Kullback-Leibler divergence $$D_\text{KL}(P \parallel Q)=\int p(x) \log\left(\frac{p(x)}{q(x)}\right)\, dx.$$
• this is in response to your request for the Bregman divergence for "general continuous distributions"; $D_F(P,Q)$ is a functional of the distributions $P$ and $Q$, so presume you could call it a"functional divergence". Jan 4 at 9:55