7
$\begingroup$

The convex conjugate of a function, say, $f:X\to \mathbb{R}$ is a function $f^*:X^*\to \mathbb{R}$ defined as $$f^*(x^*):=\sup_{x\in X} ~\langle x, x^*\rangle-f(x),$$ where $X^*$ is the topological dual of $X$ and $\langle\cdot, \cdot\rangle$ is dual pairing between $X$ and $X^*$.

The relative entropy (aka Kullback-Leibler divergence) $D(\cdot||Q):\mathcal{P}(X)\to \mathbb{R}^+$, is defined for two probability measures $P$ and $Q$ $(P\ll Q)$ as $$D(P||Q)=\int_XdP \log\frac{dP}{dQ}.$$

I have been trying to calculate the convex conjugate of map $P\mapsto D(P||Q)$ but I have failed. I know that the answer is $\log \mathbb{E}_{Q}[e^{f}]$ where $\mathbb{E}_Q[\cdot]$ is the expectation operator with respect to probability measure $Q$.

$\endgroup$
2
  • 3
    $\begingroup$ How do you "know" that is the answer? Does this come from a paper? from a book? $\endgroup$
    – Yemon Choi
    Jan 21, 2015 at 1:28
  • 1
    $\begingroup$ @YemonChoi, I have seen in many papers and books but could not find how to show that. For example, one way to prove the Donsker-Vardhan representation of relative entropy is to use Fenchel-Young inequality for which one needs to obtain convex conjugate of relative entropy. $\endgroup$ Jan 21, 2015 at 15:01

1 Answer 1

2
$\begingroup$

If you want to get it directly from the source, corollary 3 from Rockafellar's Integrals which are convex functionals I will help you find the conjugate of $D(\cdot \| \nu)$, which is a convex integral functional.

You might also consult Proposition 4.2.4 of Optimal Bounds between f-Divergences and Integral Probability Metrics by Agrawal and Horel, which states that the convex conjugate of $D(\cdot \| \nu)$ (in their notation this is $D_{\phi, \nu}$ for $\phi \colon \mathbb R \to \mathbb R \cup \{ \infty \}$, $t \mapsto t \ln(t) - t + 1 + \iota_{[0, \infty)}(t)$ which is proper, convex and lower semicontinuous with $\phi(1) = 0$ and where $\iota$ is the convex indicator function and $0 \ln(0) := 0$) is $$ Y \ni h \mapsto \int_{\mathcal X} \phi^*(h(x)) \; \text{d}\nu(x), $$ where $\phi^* \colon \mathbb R \to \mathbb R$, $s \mapsto e^{s} - 1$ is the convex conjugate of $\phi$.

In the notation of the paper mentioned we have choosen $X := M(\mathcal X)$, where $\mathcal X$ is the space from the question, and $Y := X^* \cong C(\mathcal X; \mathbb R)$ (equipped with the appropriate topologies). Note futhermore that $$\phi'(\infty) = \lim_{t \to \infty} \frac{1}{t} \phi(t) = \lim_{t \to \infty} \ln(t) - 1 + \frac{1}{t} = \infty = - \phi'(- \infty) := - \lim_{t \to - \infty} \frac{1}{t} \phi(t).$$

In the end we arrive at $$D(\cdot \| \nu)^*(h) = \int_{\mathcal X} e^{h(x)} \, \text{d}\nu(x) - 1.$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.