I know that KL divergence is a form of Bregman divergence for multinomial distributions. Is there some derivation of KL divergence from functional Bregman divergence or some generalized Bregman divergence for general continuous distributions?
1 Answer
Start from the definition of the Bregman divergence, $$D_F(P,Q)=F(P)F(Q)\langle F'(Q), PQ\rangle.$$ Substitute $F(p)=\int dx\,p(x)\log p(x)$ and work it out, $$D_F(P,Q)=\int dx\,p(x)\log p(x)\int dx\,q(x)\log q(x)\langle I+\log Q,PQ\rangle$$ $$\qquad=\int dx\,p(x)\log p(x)\int dx\,q(x)\log q(x)\int dx\,\bigl(1+\log q(x)\bigr)\bigl(p(x)q(x)\bigr)$$ $$\qquad=\int dx\,p(x)\log p(x)\int dx\,p(x)\log q(x),$$ which is the definition of the KullbackLeibler divergence $$D_\text{KL}(P \parallel Q)=\int p(x) \log\left(\frac{p(x)}{q(x)}\right)\, dx.$$

$\begingroup$ In the paper "Clustering with Bregman Divergence", I see that Bregman divergence is defined between vectors rather than continuous functions. I just read the paper "Functional Bregman divergence" and not very familiar with this concept. I see that Functional Bregman divergence is defined between functions using Frechet derivatives. The authors only derive KL divergence between simple functions which can be written as finite linear combination of indicator functions. Do you refer to functional Bregman divergence? $\endgroup$ Jan 4 at 7:16

$\begingroup$ this is in response to your request for the Bregman divergence for "general continuous distributions"; $D_F(P,Q)$ is a functional of the distributions $P$ and $Q$, so presume you could call it a"functional divergence". $\endgroup$ Jan 4 at 9:55