@stollem Well obviously I'm a lot easier to convince when the result is correct ;) Did you find the mistake in your earlier attempts?

@stollenm That sounds a little more promising. Let me know when you've tested it and I'll have a look.

@stollenm I take that to mean no, you haven't checked. Get back to me when you have.

@stollenm It "should" work? Have you tried it? Does it give the same answer for $\frac{d\operatorname{vech}A}{d\operatorname{vech}P}$?

@stollenm I don't think so? The first $DL$ is transposed.

Hi @Dorian, it's been a while since I've looked at this but unless I've missed something $dL^T = (dL)^T$ as you suggest. Note that $\frac{\partial L}{\partial A}$ is a fourth-order tensor (and not a particularly nice one) which I think is why the chain rule isn't used directly here.

Using that same paper, we can write $Q=\frac12(DL)^\top DL$ since $D^\top D = (L(I+K)L^\top)^{-1}$.

For a lower triangular matrix $A$, $\operatorname{vec}A = L^\top\operatorname{vech}A$. Ref: The Elimination Matrix: Some Lemmas and Applications.

I can generalise that result to give me $\frac{d\operatorname{vech}A}{d\operatorname{vech}P} = L(I\otimes A)Q(A^{-1}\otimes A^{-1})D$, where $Q$ is the diagonal matrix that gives us $Q\operatorname{vec}M = \operatorname{vec}(\Phi(M))$, and $D$ is the usual duplication matrix for symmetric matrices. This is more direct, and inverts $A$ rather than the larger matrix, but needs us to introduce the function $\Phi$ and matrix $Q$.

Theorem A.1 of Simo Särkkä's Bayesian Filtering and Smoothing gives a result for the scalar derivative $\frac{\partial A}{\partial\theta} = A\Phi\left(A^{-1}\frac{\partial P}{\partial\theta}{}A^{-\top}\right)$, where $P=AA^{\top}$, and $\Phi_{ij}(M) = M_{ij}$ where $i > j$; $\tfrac12M_{ij}$ where $i=j$; and $0$ where $i < j$. Using this I get the same thing as using your result above.

Columns of $L$ are either basis vectors or zero vectors. The submatrix of $L^{\top}L$ corresponding to non-zero elements of $M$ is the identity and the rest is zeros.

I prefer your result, it's a little more direct. I wonder if you should change $D$ to $L^{\top}$ though, it took me a bit of thinking before I realised that $D$ isn't the full duplication matrix.

I think the paper is here, but it appears to talk about the eigendecomposition, not the Cholesky.