Skip to main content
1 of 2
Iosif Pinelis
  • 127.8k
  • 8
  • 107
  • 229

$\newcommand\th\theta$Given a family of pdf's $p_\th$ and a prior pdf $g$, the maximizer of $$F(q,g)(y):=E_q\ln p_\th(y)-D_{KL}(q\|g)$$ is always unique -- if any two pdf's differing only on a set of measure $0$ are considered to be the same.

Indeed, letting $p(y):=\int d\th\,g(\th)p_\th(y)$ and $p(\th|y):=g(\th)p_\th(y)/p(y)$, we have \begin{align} F(q,g)(y)&=\int d\th\,q(\th)\ln p_\th(y)-\int d\th\,q(\th)\ln\frac{q(\th)}{g(\th)} \\ &=\int d\th\,q(\th)\ln\frac{g(\th)p_\th(y)}{q(\th)} \\ &=\int d\th\,q(\th)\ln\frac{p(\th|y)p(y)}{q(\th)} \\ &=\ln p(y)+\int d\th\,q(\th)\ln\frac{p(\th|y)}{q(\th)} \\ &=\ln p(y)-D_{KL}(q\|p(\cdot|y)). \end{align} So, any maximizer of $F(q,g)(y)$ in $q$ is a minimizer of $D_{KL}(q\|p(\cdot|y))$ in $q$ (and vice versa), and the only minimizer of the latter Kullback--Leibler divergence is the posterior pdf $p(\cdot|y)$.

Iosif Pinelis
  • 127.8k
  • 8
  • 107
  • 229