Timeline for Does this KL divergence inequality hold?
Current License: CC BY-SA 4.0
17 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Sep 24, 2023 at 5:36 | comment | added | Jiacai Liu | Thanks for your help. Here is a counter-example provided by Iosif Pinelis : p=(1/2,1/2) q= (1/100,99/100), \beta=0.1 | |
| Sep 24, 2023 at 5:29 | vote | accept | Jiacai Liu | ||
| Sep 24, 2023 at 5:20 | history | edited | Jiacai Liu | CC BY-SA 4.0 |
deleted 4 characters in body
|
| Sep 24, 2023 at 5:07 | history | edited | Jiacai Liu | CC BY-SA 4.0 |
added 4 characters in body
|
| Sep 24, 2023 at 2:05 | answer | added | Iosif Pinelis | timeline score: 2 | |
| Sep 23, 2023 at 21:23 | comment | added | Dabed | $ \mathrm{KL}(y||q)\\ =\sum_{x=1}^n\frac{q(x)^{1-\beta}p(x)^\beta}{\sum_{x=1}^n q(x)^{1-\beta}p(x)^\beta}\ln \frac{q(x)^{1-\beta}p(x)^\beta}{\sum_{x=1}^n q(x)^{1-\beta}p(x)^\beta}\\ \ge \sum_{x=1}^n q(x)^{1-\beta}p(x)^\beta\ln q(x)^{1-\beta}p(x)^\beta\\ \ge \sum_{x=1}^n q(x)p(x)\ln q(x)^{1-\beta}p(x)^\beta\\ =\sum_{x=1}^n (q(x)p(x)\ln p(x)^\beta+(1-\beta)q(x)p(x)\ln q(x))\qquad (q(x)<1\Rightarrow \ln q(x)<0)\\ \ge \sum_{x=1}^n q(x)p(x)\ln p(x)^\beta\\ =\beta\sum_{x=1}^n q(x)p(x)\ln p(x)\\ =\beta\sum_{x=1}^n q(x)p(x)\ln \frac{q(x)p(x)}{q(x)}\\ =\beta\mathrm{KL}(pq||q)\\ $ | |
| Sep 23, 2023 at 21:23 | comment | added | Dabed | Couldn't prove it but in case is useful I got that $\mathrm{KL}(y||q)\ge\beta\mathrm{KL}(pq||q)$, Wikipedia list the next form of the Hölder's ineq: $ (\sum_{k=1}^n |x_k|^r|y_k|^s)^{r+s}\le(\sum_{k=1}^n |x_k|^{r+s})^{r}( \sum_{k=1}^n |y_k|^{r+s})^{s}$ for $(r,s)\in\mathbb{R}_+, $ setting $x_k=q(x),y_k=p(x),r=1-\beta,s=\beta$ we get, $ \sum_{x=1}^n q(x)^{1-\beta}p(x)^\beta\le(\sum_{x=1}^n q(x))^{1-\beta}( \sum_{x=1}^n p(x))^{1-\beta}=1\cdot 1=1 \Rightarrow\frac{q(x)^{1-\beta}p(x)^\beta}{\sum_{x=1}^n q(x)^{1-\beta}p(x)^\beta}\ge q(x)^{1-\beta}p(x)^\beta $ Therefore: | |
| Sep 23, 2023 at 20:56 | history | edited | Michael Hardy | CC BY-SA 4.0 |
added 16 characters in body
|
| Sep 23, 2023 at 17:07 | comment | added | Mark Schultz-Wu | Also note that your KL between a distribution and a geometric mixture has been studied in other places, e.g. here, where it is computed for geometric mixtures of Gaussians. I think the content of that paper is enough to check your inequality for Gaussians. I would recommend doing this (it should be quite quick), as it may be sufficient to produce a quick counterexample to your inequality. | |
| Sep 23, 2023 at 17:05 | comment | added | Mark Schultz-Wu | Inequalities close to this hold, but I don't have time to check this one. See Annealing Between Distributions by Averaging Moments. They show that the distribution $y$ that minimizes $$f(y) :=(1-\beta)\mathsf{KL}(y||q) +\beta \mathsf{KL}(y||p)$$ takes the form $y\propto q^{1-\beta}p^\beta$. We then have that $$f(q) \geq f(y)\implies \beta \mathsf{KL}(q||p) \geq (1-\beta)\mathsf{KL}(y||q)+\beta\mathsf{KL}(y||p).$$ I don't know if a modified argument can get your inequality. | |
| Sep 23, 2023 at 9:42 | history | edited | gmvh | CC BY-SA 4.0 |
Corrected grammar
|
| Sep 23, 2023 at 7:33 | history | edited | Jiacai Liu | CC BY-SA 4.0 |
added 1 character in body
|
| Sep 23, 2023 at 7:33 | history | edited | Jiacai Liu | CC BY-SA 4.0 |
added 1 character in body; edited title
|
| Sep 23, 2023 at 4:56 | history | edited | David Roberts♦ | CC BY-SA 4.0 |
minor edits
|
| Sep 23, 2023 at 3:09 | history | edited | Jiacai Liu | CC BY-SA 4.0 |
added 3 characters in body
|
| S Sep 23, 2023 at 3:08 | review | First questions | |||
| Sep 23, 2023 at 9:42 | |||||
| S Sep 23, 2023 at 3:08 | history | asked | Jiacai Liu | CC BY-SA 4.0 |