Looking for a reference: $f$-divergences are lower semicontinuous

Question

I know that the weak lower semi-continuity of the KL divergence was proved in [1]. If I remember well, the same property is true for any $f$ divergence (with suitable assumptions on the probability space). I am looking for some reference about it.

[1] Posner, Random Coding Strategies for Minimum Entropy, 1975.

Edit.
Here is what I believe a standard definition of $f$ divergences, which includes the case of measures not absolutely continuous to each other. This definition is taken from http://people.lids.mit.edu/yp/homepage/data/LN_fdiv.pdf

Definition 7.1. Let $f:(0,\infty)\to\mathbb R$ be a convex function with $f(1)=0$. Let $P$ and $Q$ be two probability distributions on a measurable space $(\mathcal X, \mathcal F)$. If $P\ll Q$ then the $f$-divergence is defined as $$D_f(P\|Q)=\mathbb E_Q[f(dP/dQ)]$$ where $dP/dQ$ is the Radon-Nikodym derivative and $f(0)=f(0+)$. More generally, let $f'(\infty)=\lim_{x\to 0}xf(1/x)$. Let $R$ be such that $Q\ll R$ and $P\ll R$ (such an $R$ always exists, for instance take $R=\frac{1}{2}(P+Q)$. Then we have $$D_f(P\|Q) = f'(\infty)P(dQ/dR=0)+\int_{dQ/dR>0}\frac{dQ}{dR}f\left(\frac{dP/dR}{dQ/dR}\right)dR\,,$$ with the agreement that if $P(dQ/dR=0)=0$ the last term is taken to be zero regardless of the value of $f'(\infty)$ (which could be infinite).

Answer 1

I you further assume that $f$ is lower semicontinuous then the $f$ divergence is weakly lsc w.r.t. to both its primary argument $P$ and reference measure $Q$, i-e $$ D_f(P\| Q)\leq \liminf\limits_{n\to\infty} D_f(P_n\|Q_n) $$ as soon as $P_n\rightharpoonup P$ and $Q_n\rightharpoonup Q$. For a reference see e.g. theorem 2.34 pp. 65 in [1]. The lower semicontinuity of $f$ is actually necessary.

---

[1] Ambrosio, Luigi, Nicola Fusco, and Diego Pallara. Functions of bounded variation and free discontinuity problems. Courier Corporation, 2000.

Answer 2

This can be seen analogous as for the KL divergence using a duality representation. For the case, where $f'(\infty)=\infty$, i.e. $f$ has super linear growth, we can define thanks to the convexity of $f$ it Legendre-Fenchel dual by $$ f^*(r) := \sup_{s>0} \{ r\,s - f(r) \} . $$ Note, that for $f(s)= s \log s - s +s$ (KL-divergence), it holds $f^*(r) = e^r -1$. With this, we find the dual representation $$ D_f(P\| Q) = \sup \left\{ \int g \, dP - \int f^*\circ g \, dQ : g \in L^\infty(P +Q) \right\}. $$ Then, by a density argument, we can restrict the optimization to $C_b$-functions and it also holds $$ D_f(P\| Q) = \sup_{g\in C_b(X)} \left\{ \int g \, dP - \int f^*\circ g \, dQ \right\}. $$ In this form, the weak lower semicontinuity is clear, since it holds for any fixed $g\in C_b(X)$ and hence also holds for the $\sup$.

References

Dualization in convex analysis is pretty standard and classic sources from Rockafellar should contain similar results along this lines. For instance

• R. T. Rockafellar, Integrals which are convex functionals, Pacific J. Math. 24, no. 3 (1968), 525-539.

The paper

• Broniatowski, M., & Keziou, A. (2006). Minimization of φ-divergences on sets of signed measures, Studia Scientiarum Mathematicarum Hungarica, 43(4), 403-442. arXiv:1003.5457

contains in Section 4 an overview with some more historic references.

Answer 3

I think the most general reference is:

M. Liero, A. Mielke, and G. Savaré. Optimal entropy-transport problems and a newhellinger–kantorovich distance between positive measures. Inventiones mathematicae, 211, 2018.
arxiv:1508.07941

Corollary 2.9 proves the lower semicontinuity wrt to the narrow topology under very general assumptions, and Remark 2.1 shows that the weak topology and the narrow topology coincide if $\mathcal X$ is Polish.