Skip to main content
added 107 characters in body
Source Link
Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229

$\newcommand{\R}{\mathbb R}$$\newcommand{\R}{\mathbb R}\newcommand{\KL}{{\operatorname{KL}}}$For $j=1,2$, let $P_j:=N(\mu_j,I_d)$, where $\mu_2=\mu_1+v$ and $v$ is a unit vector. So, for the pdf's $p_j$ of $P_j$ we have \begin{equation} p_j(x)=(2\pi)^{-d/2} e^{-|x-\mu_j|^2/2} \end{equation}\begin{equation*} p_j(x)=(2\pi)^{-d/2} e^{-|x-\mu_j|^2/2} \end{equation*} for all $x\in\R^d$, where $|\cdot|$ is the Euclidean norm. Let also $\cdot$ denote the dot product.

Then the KL divergence between $P_1$ and $P_2$ is \begin{equation} \begin{aligned} KL(P_1,P_2)&=\int_{\R^d}p_1\ln\frac{p_1}{p_2} \\ &=\int_{\R^d}dx\,p_1(x)\,\tfrac12(|x-\mu_2|^2-|x-\mu_1|^2) \\ &=\int_{\R^d}dx\,p_1(x)\,\big((\mu_1-\mu_2)\cdot x+\tfrac12(|\mu_2|^2-|\mu_1|^2)\big) \\ &=(\mu_1-\mu_2)\cdot\mu_1+\tfrac12(|\mu_2|^2-|\mu_1|^2) \\ &=(\mu_1-\mu_2)\cdot\mu_1-\tfrac12(\mu_1-\mu_2)\cdot(\mu_1+\mu_2) \\ &=\tfrac12\,(\mu_1-\mu_2)\cdot(\mu_1-\mu_2)=\tfrac12\,(-v)\cdot(-v)=\tfrac12. \end{aligned} \end{equation}\begin{equation*} \begin{aligned} \KL(P_1,P_2)&=\int_{\R^d}p_1\ln\frac{p_1}{p_2} \\ &=\int_{\R^d}dx\,p_1(x)\,\tfrac12(|x-\mu_2|^2-|x-\mu_1|^2) \\ &=\int_{\R^d}dx\,p_1(x)\,\big((\mu_1-\mu_2)\cdot x+\tfrac12(|\mu_2|^2-|\mu_1|^2)\big) \\ &=(\mu_1-\mu_2)\cdot\mu_1+\tfrac12(|\mu_2|^2-|\mu_1|^2) \\ &=(\mu_1-\mu_2)\cdot\mu_1-\tfrac12(\mu_1-\mu_2)\cdot(\mu_1+\mu_2) \\ &=\tfrac12\,(\mu_1-\mu_2)\cdot(\mu_1-\mu_2)=\tfrac12\,(-v)\cdot(-v)=\tfrac12. \end{aligned} \end{equation*}

ThusTherefore and because $V$ is a unit random vector, the expected KL divergence between $P_1$$N(\mu_1,I_d)$ and $P_2$$N(\mu_1+V,I_d)$ is $\tfrac12$ as well. : \begin{equation*} \mathsf E\,\KL\big(N(\mu_1,I_d),N(\mu_1+V,I_d)\big)=\tfrac12. \end{equation*} (The condition that $V$ is uniformly distributed and orthogonal to $\mu_1$ was not used here; we have onlyneeded or used the condition that the random vector $V$ is unithere.)

$\newcommand{\R}{\mathbb R}$For $j=1,2$, let $P_j:=N(\mu_j,I_d)$, where $\mu_2=\mu_1+v$ and $v$ is a unit vector. So, for the pdf's $p_j$ of $P_j$ we have \begin{equation} p_j(x)=(2\pi)^{-d/2} e^{-|x-\mu_j|^2/2} \end{equation} for all $x\in\R^d$, where $|\cdot|$ is the Euclidean norm. Let also $\cdot$ denote the dot product.

Then the KL divergence between $P_1$ and $P_2$ is \begin{equation} \begin{aligned} KL(P_1,P_2)&=\int_{\R^d}p_1\ln\frac{p_1}{p_2} \\ &=\int_{\R^d}dx\,p_1(x)\,\tfrac12(|x-\mu_2|^2-|x-\mu_1|^2) \\ &=\int_{\R^d}dx\,p_1(x)\,\big((\mu_1-\mu_2)\cdot x+\tfrac12(|\mu_2|^2-|\mu_1|^2)\big) \\ &=(\mu_1-\mu_2)\cdot\mu_1+\tfrac12(|\mu_2|^2-|\mu_1|^2) \\ &=(\mu_1-\mu_2)\cdot\mu_1-\tfrac12(\mu_1-\mu_2)\cdot(\mu_1+\mu_2) \\ &=\tfrac12\,(\mu_1-\mu_2)\cdot(\mu_1-\mu_2)=\tfrac12\,(-v)\cdot(-v)=\tfrac12. \end{aligned} \end{equation}

Thus, the expected KL divergence between $P_1$ and $P_2$ is $\tfrac12$ as well. (The condition that $V$ is uniformly distributed and orthogonal to $\mu_1$ was not used here; we have only used the condition that the random vector $V$ is unit.)

$\newcommand{\R}{\mathbb R}\newcommand{\KL}{{\operatorname{KL}}}$For $j=1,2$, let $P_j:=N(\mu_j,I_d)$, where $\mu_2=\mu_1+v$ and $v$ is a unit vector. So, for the pdf's $p_j$ of $P_j$ we have \begin{equation*} p_j(x)=(2\pi)^{-d/2} e^{-|x-\mu_j|^2/2} \end{equation*} for all $x\in\R^d$, where $|\cdot|$ is the Euclidean norm. Let also $\cdot$ denote the dot product.

Then the KL divergence between $P_1$ and $P_2$ is \begin{equation*} \begin{aligned} \KL(P_1,P_2)&=\int_{\R^d}p_1\ln\frac{p_1}{p_2} \\ &=\int_{\R^d}dx\,p_1(x)\,\tfrac12(|x-\mu_2|^2-|x-\mu_1|^2) \\ &=\int_{\R^d}dx\,p_1(x)\,\big((\mu_1-\mu_2)\cdot x+\tfrac12(|\mu_2|^2-|\mu_1|^2)\big) \\ &=(\mu_1-\mu_2)\cdot\mu_1+\tfrac12(|\mu_2|^2-|\mu_1|^2) \\ &=(\mu_1-\mu_2)\cdot\mu_1-\tfrac12(\mu_1-\mu_2)\cdot(\mu_1+\mu_2) \\ &=\tfrac12\,(\mu_1-\mu_2)\cdot(\mu_1-\mu_2)=\tfrac12\,(-v)\cdot(-v)=\tfrac12. \end{aligned} \end{equation*}

Therefore and because $V$ is a unit random vector, the expected KL divergence between $N(\mu_1,I_d)$ and $N(\mu_1+V,I_d)$ is $\tfrac12$ as well: \begin{equation*} \mathsf E\,\KL\big(N(\mu_1,I_d),N(\mu_1+V,I_d)\big)=\tfrac12. \end{equation*} (The condition that $V$ is uniformly distributed and orthogonal to $\mu_1$ was not needed or used here.)

Source Link
Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229

$\newcommand{\R}{\mathbb R}$For $j=1,2$, let $P_j:=N(\mu_j,I_d)$, where $\mu_2=\mu_1+v$ and $v$ is a unit vector. So, for the pdf's $p_j$ of $P_j$ we have \begin{equation} p_j(x)=(2\pi)^{-d/2} e^{-|x-\mu_j|^2/2} \end{equation} for all $x\in\R^d$, where $|\cdot|$ is the Euclidean norm. Let also $\cdot$ denote the dot product.

Then the KL divergence between $P_1$ and $P_2$ is \begin{equation} \begin{aligned} KL(P_1,P_2)&=\int_{\R^d}p_1\ln\frac{p_1}{p_2} \\ &=\int_{\R^d}dx\,p_1(x)\,\tfrac12(|x-\mu_2|^2-|x-\mu_1|^2) \\ &=\int_{\R^d}dx\,p_1(x)\,\big((\mu_1-\mu_2)\cdot x+\tfrac12(|\mu_2|^2-|\mu_1|^2)\big) \\ &=(\mu_1-\mu_2)\cdot\mu_1+\tfrac12(|\mu_2|^2-|\mu_1|^2) \\ &=(\mu_1-\mu_2)\cdot\mu_1-\tfrac12(\mu_1-\mu_2)\cdot(\mu_1+\mu_2) \\ &=\tfrac12\,(\mu_1-\mu_2)\cdot(\mu_1-\mu_2)=\tfrac12\,(-v)\cdot(-v)=\tfrac12. \end{aligned} \end{equation}

Thus, the expected KL divergence between $P_1$ and $P_2$ is $\tfrac12$ as well. (The condition that $V$ is uniformly distributed and orthogonal to $\mu_1$ was not used here; we have only used the condition that the random vector $V$ is unit.)