A random variable $X$ is said to be sub-gaussian with mean $\mu$ and pseudo-variance $\sigma^2$ iff $$\mathbb \log(E[\exp(t(X-\mu))]) \le \frac{t^2}{2\sigma^2},\;\forall t \in \mathbb R. $$

It's a standard computation that the KL divergence between two random variables with means $\mu_1$ and $\mu_2$ and same variance $\sigma^2 > 0$ is $\dfrac{(\mu_1 - \mu_2)^2}{2\sigma^2}$.


What is good upper-bound for the KL divergence between two sub-gaussian variables with means $\mu_1$ and $\mu_2$ and same pseudo-variance $\sigma^2 > 0$, and full support ?

  • 1
    $\begingroup$ I don't think there is one because of the usual support issue. One thing you might be able to do, though, is to post-process the two distributions in question, say, $P,Q$ for $D(P\| Q)$ by convolution with a $\varepsilon$-variance Gaussian (like an addition of a small amount of independent noise). I believe the KL divergence topology, allowing these perturbations, is equivalent to the weak topology for probability distributions. $\endgroup$ – enthdegree Jul 31 '18 at 19:16
  • $\begingroup$ Thanks for the response. Hum, I'm not sure this can work. The weak topology is generated by the Wasserstein $W_1$ distance which is uniformly upper-bounded by on probability measures on a bounded set $A$ by $\operatorname{diam}(A) \sqrt{\operatorname{KL}/2}$, but what I need is an upper bound on KL. Except I'm missing something here. $\endgroup$ – dohmatob Jul 31 '18 at 19:52
  • $\begingroup$ @enthdegree If it helps you may assume that both variables have full support. $\endgroup$ – dohmatob Jul 31 '18 at 19:59
  • $\begingroup$ Also note that by data-processing inequalities, $\operatorname{KL}(P\| Q) \ge \operatorname{KL}(P * R \| Q * R)$, and so the convolution trick you mention won't help get an upper bound on $\operatorname{KL}(P\| Q)$. $\endgroup$ – dohmatob Jul 31 '18 at 20:10
  • $\begingroup$ An upper bound with only the assumptions you've stated cannot be expected in general. The "support issue" I was getting at is worse than just that of proper supports: For $\log(dP/dQ)$ to have finite measure wrt $P$, then (almost by tautology) $Q$'s tail should ($P$-)usually not be very small in proportion to $P$'s tail. Sub-Gaussianity of the two variables only says that their two tails are dominated by some Gaussian's tail, and says nothing about the relations of the tails to each other which is what you need. $\endgroup$ – enthdegree Jul 31 '18 at 20:37

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.