Take the 2-minute tour ×
MathOverflow is a question and answer site for professional mathematicians. It's 100% free, no registration required.

Note: Moved to math.stackexchange.com. Sorry for the off-topic question!

[Context: I'm working through problem set one of Andrew Ng's Machine Learning course, question 2, trying to derive the Hessian myself. My trouble, though, is with the simple linear algebra. I'll ignore the regularization parameters in my question to keep things simple.]

I can find $H_{jk}$ by taking partial derivatives of the original function with respect to $j$ and $k$. I've worked it through to the point where I know that $H_{jk} = \sum_{i=1}^m d_i X_{ij} X_{ik}$, and I know the answer is that $H = X^TDX$.

($d$ is a vector of reals, $H$, $X$ and $D$ are $m \times m$ matrices, $D$ diagonal with $D_{ii} = d_i$.)

It's easy for me to verify that $H = X^TDX$ gives $H_{jk} = \sum_{i=1}^m d_i X_{ij} X_{ik}$. But I don't "see" how to go easily in the opposite direction.

1) Is there a method, or a set of rules to memorize, or a way of talking through it that would make it easy, or do I just need to do enough basic linear algebra homework to start to recognize patterns?

2) Am I doing it wrong by breaking it down and deriving $H_{jk}$? Are there tricks for taking these kind of derivatives (which involve logs and exponentials) over the whole matrix at once?

share|improve this question
This question is not quite appropriate for MO; you might have better luck at math.stackexchange.com. –  Qiaochu Yuan Feb 25 '11 at 19:52
add comment

1 Answer

I'm not sure, but it seems to me you're done!: $H_{jk}=\sum_{i=1}^m d_i X_{ij} X_{ik}=\sum_{i=1}^m X^T_{ji} D_{ii} X_{ik}$. Where, as you mention $D_{ii}=d_i$, is a diagonal matrix, therefore: $H_{jk} =\sum_{i=1}^m \sum_{l=1}^m \sum_{n=1}^m X^T_{jl} D_{ln} X_{nk}=X^TDX$.

share|improve this answer
add comment

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.