Derivative of an integral of a Gaussian

Question

I'd like to compute the derivative of an expected value w.r.t one of the parameters that define the mean of a Gaussian: $ Z=\int \mathcal{N}(x;\mu,\Sigma)f(x) \, dx $, then $ \frac{dZ}{dK}=\text{??}$ when $\mu=g(K)\in\mathbb{R}^{n\times 1}$. I have three problems with this: 1. The dimensions are not adding up for me. 2. How can I compute this also numerically if $g(K)$ gets too complicated? 3. How do I do this if $\Sigma=h(K)\in\mathbb{R}^{n\times n}$, too?

1. This is what I have so far, if I ignore the dependency of $\mu$ in $K$ for a minute: $$ \frac{dZ}{d\mu} = \int \frac{d\mathcal{N}(x;\mu,\Sigma)}{d\mu}f(x)\,dx $$ and since $\mathcal{N}(x;\mu,\Sigma)=(2\pi)^{-n/2} \det(\Sigma)^{-1} \exp(-0.5(x-\mu)^T\Sigma^{-1}(x-\mu))$, we have that $\frac{d\mathcal{N}(x;\mu,\Sigma)}{d\mu}=\mathcal{N}(x;\mu,\Sigma)\frac{d\ln\mathcal{N}(x;\mu,\Sigma)}{d\mu}$ so now we can get: $$ \frac{dZ}{d\mu} = \int \mathcal{N}(x;\mu,\Sigma)\frac{d\ln\mathcal{N}(x;\mu,\Sigma)}{d\mu} f(x) \, dx = \mathbb{E}\left[\frac{d\ln\mathcal{N}(x;\mu,\Sigma)}{d\mu}\right]$$ This means I can compute $$ \frac{dZ}{d\mu} = \mathbb{E}\left[\frac{d\ln\mathcal{N}(x;\mu,\Sigma)}{d\mu}\right] = \mathbb{E}\left[\Sigma^{-1}(x-\mu)\right]$$ Now, comes the problem - this expected value is going to be a vector $\in \mathbb{R}^{n\times 1}$. Now if I want to derive the gradient w.r.t $K$ then: $$ \frac{dZ}{dK} = \frac{dZ}{d\mu}\frac{d\mu}{dK}$$ and lets say that $\mu$ is the result of calculations of $K\in\mathbb{R}^{p\times q}$, what are the dimensions of $\frac{d\mu}{dK}$? Even if I got the ordering or transpose wrong in the matrix derivations, I still don't see how that dimensions should work because I expect that $ \frac{dZ}{dK} \in\mathbb{R}^{p\times q}$. An example for $\mu$ can be: $$ \mu = \left[ AKv ; (AK)^2v; (AK)^3v ; \ldots \right] \in \mathbb{R}^{n\times 1} $$ where the constant $A\in\mathbb{R}^{q\times p}, v\in\mathbb{R}^{q\times 1}$.

2. Is there any software that can compute this symbolically? Or numerically? Matlab will not derive a vector by a matrix. I'm trying to do this with PyTorch in Python with the autograd, but the problem is again the dimensions, I'm not sure what to put in the vector $v$ that multiplies the gradient.

3. Now let's suppose that $K$ also affects $\Sigma$ too, how do I compute the total influence (gradient) of $K$ on $Z$? With the technique above I can also compute: $$ \frac{dZ}{d\Sigma} = \mathbb{E}\left[\frac{d\ln\mathcal{N}(x;\mu,\Sigma)}{d\Sigma}\right] = \mathbb{E}\left[\Sigma^{-1}(x-\mu)(x-\mu)^T\Sigma^{-1}-\Sigma^{-1}\right] $$

Answer 1

Q1: $dZ/dK$ is a $p\times q$ matrix with elements $$\bigl[dZ/dK\bigr]_{ij}=\sum_{k=1}^n\frac{\partial \mu_k}{\partial K_{ij}}\,\mathbb{E}[f(x)\,\Sigma^{-1}\cdot(x-\mu)]_k.$$ Q2: To evaluate this you will have to specify the function $f(x)$. For some $f$ you will be able to evaluate the expectation value symbolically, but in general a numerical evaluation should be easy and accurate because of the Gaussian weight factor.
Q3: If $\Sigma$ also depends on $K$ you add the terms $\sum_{kl}(\partial\Sigma_{kl}/\partial K_{ij})(\partial Z/\partial\Sigma_{kl})$.