Skip to main content
2 of 2
added 46 characters in body
Michael Hardy
  • 1
  • 12
  • 85
  • 126

$\newcommand\th\theta$Given a family of pdf's $p_\th$ and a prior pdf $g$, the maximizer of $$F(q,g)(y):=E_q\ln p_\th(y)-D_{KL}(q\parallel g)$$ is always unique -- if any two pdf's differing only on a set of measure $0$ are considered to be the same.

Indeed, letting $p(y):=\int d\th\,g(\th)p_\th(y)$ and $p(\th\mid y):=g(\th)p_\th(y)/p(y)$, we have \begin{align} F(q,g)(y)&=\int d\th\,q(\th)\ln p_\th(y)-\int d\th\,q(\th)\ln\frac{q(\th)}{g(\th)} \\ &=\int d\th\,q(\th)\ln\frac{g(\th)p_\th(y)}{q(\th)} \\ &=\int d\th\,q(\th)\ln\frac{p(\th\mid y)p(y)}{q(\th)} \\ &=\ln p(y)+\int d\th\,q(\th)\ln\frac{p(\th\mid y)}{q(\th)} \\ &=\ln p(y)-D_{KL}(q\parallel p(\cdot\mid y)). \end{align} So, any maximizer of $F(q,g)(y)$ in $q$ is a minimizer of $D_{KL}(q\parallel p(\cdot\mid y))$ in $q$ (and vice versa), and the only minimizer of the latter Kullback–Leibler divergence is the posterior pdf $p(\cdot\mid y)$.

Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229