KL divergence between gaussian with uniform prior

Question

I have 2 normal distributions $\mathcal{N}(\mu_1, \mathbb{I}_d)$ where $\mu_1$ is a fixed vector in $\mathbb{R}^d$ and $\mathcal{N}(\mu_2, \mathbb{I}_d)$, where $\mu_2$ is $\mu_1 + V$, where $V$ is uniformly distributed and orthogonal to $\mu_1$, that is $V^\top \mu_1 = 0$. (In 3 dimensions $V$ would be uniformly distributed in the unit circle of the $y-z$ plane if $\mu_1$ is the unit vector in the $x$ axis.) My question is what would be the KL divergence between $\mathcal{N}(\mu_1, \mathbb{I}_d)$ and $\mathcal{N}(\mu_2, \mathbb{I}_d)$?

Is the question even valid, and if so how to proceed.

Answer 1

$\newcommand{\R}{\mathbb R}\newcommand{\KL}{{\operatorname{KL}}}$For $j=1,2$, let $P_j:=N(\mu_j,I_d)$, where $\mu_2=\mu_1+v$ and $v$ is a unit vector. So, for the pdf's $p_j$ of $P_j$ we have \begin{equation*} p_j(x)=(2\pi)^{-d/2} e^{-|x-\mu_j|^2/2} \end{equation*} for all $x\in\R^d$, where $|\cdot|$ is the Euclidean norm. Let also $\cdot$ denote the dot product.

Then the KL divergence between $P_1$ and $P_2$ is \begin{equation*} \begin{aligned} \KL(P_1,P_2)&=\int_{\R^d}p_1\ln\frac{p_1}{p_2} \\ &=\int_{\R^d}dx\,p_1(x)\,\tfrac12(|x-\mu_2|^2-|x-\mu_1|^2) \\ &=\int_{\R^d}dx\,p_1(x)\,\big((\mu_1-\mu_2)\cdot x+\tfrac12(|\mu_2|^2-|\mu_1|^2)\big) \\ &=(\mu_1-\mu_2)\cdot\mu_1+\tfrac12(|\mu_2|^2-|\mu_1|^2) \\ &=(\mu_1-\mu_2)\cdot\mu_1-\tfrac12(\mu_1-\mu_2)\cdot(\mu_1+\mu_2) \\ &=\tfrac12\,(\mu_1-\mu_2)\cdot(\mu_1-\mu_2)=\tfrac12\,(-v)\cdot(-v)=\tfrac12. \end{aligned} \end{equation*}

Therefore and because $V$ is a unit random vector, the expected KL divergence between $N(\mu_1,I_d)$ and $N(\mu_1+V,I_d)$ is $\tfrac12$ as well: \begin{equation*} \mathsf E\,\KL\big(N(\mu_1,I_d),N(\mu_1+V,I_d)\big)=\tfrac12. \end{equation*} (The condition that $V$ is uniformly distributed and orthogonal to $\mu_1$ was not needed or used here.)