0
$\begingroup$

A random variable $X$ is said to be sub-gaussian with mean $\mu$ and pseudo-variance $\sigma^2$ iff $$\mathbb \log(E[\exp(t(X-\mu))]) \le \frac{t^2}{2\sigma^2},\;\forall t \in \mathbb R. $$

It's a standard computation that the KL divergence between two random variables with means $\mu_1$ and $\mu_2$ and same variance $\sigma^2 > 0$ is $\dfrac{(\mu_1 - \mu_2)^2}{2\sigma^2}$.

Question

What is good upper-bound for the KL divergence between two sub-gaussian variables with means $\mu_1$ and $\mu_2$ and same pseudo-variance $\sigma^2 > 0$, and full support ?

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ I don't think there is one because of the usual support issue. One thing you might be able to do, though, is to post-process the two distributions in question, say, $P,Q$ for $D(P\| Q)$ by convolution with a $\varepsilon$-variance Gaussian (like an addition of a small amount of independent noise). I believe the KL divergence topology, allowing these perturbations, is equivalent to the weak topology for probability distributions. $\endgroup$
    – Christian Chapman
    Commented Jul 31, 2018 at 19:16
  • $\begingroup$ Thanks for the response. Hum, I'm not sure this can work. The weak topology is generated by the Wasserstein $W_1$ distance which is uniformly upper-bounded by on probability measures on a bounded set $A$ by $\operatorname{diam}(A) \sqrt{\operatorname{KL}/2}$, but what I need is an upper bound on KL. Except I'm missing something here. $\endgroup$
    – dohmatob
    Commented Jul 31, 2018 at 19:52
  • $\begingroup$ @enthdegree If it helps you may assume that both variables have full support. $\endgroup$
    – dohmatob
    Commented Jul 31, 2018 at 19:59
  • $\begingroup$ Also note that by data-processing inequalities, $\operatorname{KL}(P\| Q) \ge \operatorname{KL}(P * R \| Q * R)$, and so the convolution trick you mention won't help get an upper bound on $\operatorname{KL}(P\| Q)$. $\endgroup$
    – dohmatob
    Commented Jul 31, 2018 at 20:10
  • $\begingroup$ An upper bound with only the assumptions you've stated cannot be expected in general. The "support issue" I was getting at is worse than just that of proper supports: For $\log(dP/dQ)$ to have finite measure wrt $P$, then (almost by tautology) $Q$'s tail should ($P$-)usually not be very small in proportion to $P$'s tail. Sub-Gaussianity of the two variables only says that their two tails are dominated by some Gaussian's tail, and says nothing about the relations of the tails to each other which is what you need. $\endgroup$
    – Christian Chapman
    Commented Jul 31, 2018 at 20:37

0

Reset to default

You must log in to answer this question.