KL divergence between gaussian with uniform prior I have 2 normal distributions $\mathcal{N}(\mu_1, \mathbb{I}_d)$ where $\mu_1$ is a fixed vector in $\mathbb{R}^d$ and $\mathcal{N}(\mu_2, \mathbb{I}_d)$, where $\mu_2$ is $\mu_1 + V$, where $V$ is uniformly distributed and orthogonal to $\mu_1$, that is $V^\top \mu_1 = 0$. (In 3 dimensions $V$ would be uniformly distributed in the unit circle of the $y-z$ plane if $\mu_1$ is the unit vector in the $x$ axis.)
My question is what would be the KL divergence between $\mathcal{N}(\mu_1, \mathbb{I}_d)$ and $\mathcal{N}(\mu_2, \mathbb{I}_d)$?
Is the question even valid, and if so how to proceed.
 A: $\newcommand{\R}{\mathbb R}\newcommand{\KL}{{\operatorname{KL}}}$For $j=1,2$, let $P_j:=N(\mu_j,I_d)$, where $\mu_2=\mu_1+v$ and $v$ is a unit vector. So, for the pdf's $p_j$ of $P_j$ we have
\begin{equation*}
    p_j(x)=(2\pi)^{-d/2} e^{-|x-\mu_j|^2/2}
\end{equation*}
for all $x\in\R^d$, where $|\cdot|$ is the Euclidean norm. Let also $\cdot$ denote the dot product.
Then the KL divergence between $P_1$ and $P_2$ is
\begin{equation*}
\begin{aligned}
    \KL(P_1,P_2)&=\int_{\R^d}p_1\ln\frac{p_1}{p_2} \\ 
    &=\int_{\R^d}dx\,p_1(x)\,\tfrac12(|x-\mu_2|^2-|x-\mu_1|^2) \\ 
    &=\int_{\R^d}dx\,p_1(x)\,\big((\mu_1-\mu_2)\cdot x+\tfrac12(|\mu_2|^2-|\mu_1|^2)\big) \\ 
        &=(\mu_1-\mu_2)\cdot\mu_1+\tfrac12(|\mu_2|^2-|\mu_1|^2) \\ 
    &=(\mu_1-\mu_2)\cdot\mu_1-\tfrac12(\mu_1-\mu_2)\cdot(\mu_1+\mu_2) \\ 
    &=\tfrac12\,(\mu_1-\mu_2)\cdot(\mu_1-\mu_2)=\tfrac12\,(-v)\cdot(-v)=\tfrac12.  
    \end{aligned}
\end{equation*}
Therefore and because $V$ is a unit random vector, the expected KL divergence between $N(\mu_1,I_d)$ and $N(\mu_1+V,I_d)$ is $\tfrac12$ as well:
\begin{equation*}
    \mathsf E\,\KL\big(N(\mu_1,I_d),N(\mu_1+V,I_d)\big)=\tfrac12. 
\end{equation*}
(The condition that $V$ is uniformly distributed and orthogonal to $\mu_1$ was not needed or used here.)
