Timeline for Upper-bound KL divergence between sub-gaussian variables with same variance
Current License: CC BY-SA 4.0
8 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 31, 2018 at 20:56 | comment | added | dohmatob | Agreed. The problem is indeed under-constrained. Nothing can be salvaged from the wreckage. Thanks. | |
| Jul 31, 2018 at 20:37 | comment | added | Christian Chapman | An upper bound with only the assumptions you've stated cannot be expected in general. The "support issue" I was getting at is worse than just that of proper supports: For $\log(dP/dQ)$ to have finite measure wrt $P$, then (almost by tautology) $Q$'s tail should ($P$-)usually not be very small in proportion to $P$'s tail. Sub-Gaussianity of the two variables only says that their two tails are dominated by some Gaussian's tail, and says nothing about the relations of the tails to each other which is what you need. | |
| Jul 31, 2018 at 20:17 | history | edited | dohmatob | CC BY-SA 4.0 |
added 18 characters in body
|
| Jul 31, 2018 at 20:10 | comment | added | dohmatob | Also note that by data-processing inequalities, $\operatorname{KL}(P\| Q) \ge \operatorname{KL}(P * R \| Q * R)$, and so the convolution trick you mention won't help get an upper bound on $\operatorname{KL}(P\| Q)$. | |
| Jul 31, 2018 at 19:59 | comment | added | dohmatob | @enthdegree If it helps you may assume that both variables have full support. | |
| Jul 31, 2018 at 19:52 | comment | added | dohmatob | Thanks for the response. Hum, I'm not sure this can work. The weak topology is generated by the Wasserstein $W_1$ distance which is uniformly upper-bounded by on probability measures on a bounded set $A$ by $\operatorname{diam}(A) \sqrt{\operatorname{KL}/2}$, but what I need is an upper bound on KL. Except I'm missing something here. | |
| Jul 31, 2018 at 19:16 | comment | added | Christian Chapman | I don't think there is one because of the usual support issue. One thing you might be able to do, though, is to post-process the two distributions in question, say, $P,Q$ for $D(P\| Q)$ by convolution with a $\varepsilon$-variance Gaussian (like an addition of a small amount of independent noise). I believe the KL divergence topology, allowing these perturbations, is equivalent to the weak topology for probability distributions. | |
| Jul 31, 2018 at 19:13 | history | asked | dohmatob | CC BY-SA 4.0 |