Take the 2-minute tour ×
MathOverflow is a question and answer site for professional mathematicians. It's 100% free, no registration required.

Note: Moved to math.stackexchange.com. Sorry for the off-topic question!

[Context: I'm working through problem set one of Andrew Ng's Machine Learning course, question 2, trying to derive the Hessian myself. My trouble, though, is with the simple linear algebra. I'll ignore the regularization parameters in my question to keep things simple.]

I can find $H_{jk}$ by taking partial derivatives of the original function with respect to $j$ and $k$. I've worked it through to the point where I know that $H_{jk} = \sum_{i=1}^m d_i X_{ij} X_{ik}$, and I know the answer is that $H = X^TDX$.

($d$ is a vector of reals, $H$, $X$ and $D$ are $m \times m$ matrices, $D$ diagonal with $D_{ii} = d_i$.)

It's easy for me to verify that $H = X^TDX$ gives $H_{jk} = \sum_{i=1}^m d_i X_{ij} X_{ik}$. But I don't "see" how to go easily in the opposite direction.

1) Is there a method, or a set of rules to memorize, or a way of talking through it that would make it easy, or do I just need to do enough basic linear algebra homework to start to recognize patterns?

2) Am I doing it wrong by breaking it down and deriving $H_{jk}$? Are there tricks for taking these kind of derivatives (which involve logs and exponentials) over the whole matrix at once?

share|improve this question

put on hold as off-topic by Ricardo Andrade, Yemon Choi, Stefan Kohl, Jeremy Rouse, Chris Godsil Nov 26 at 3:35

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "MathOverflow is for mathematicians to ask each other questions about their research. See Math.StackExchange to ask general questions in mathematics." – Ricardo Andrade, Yemon Choi, Stefan Kohl, Jeremy Rouse, Chris Godsil
If this question can be reworded to fit the rules in the help center, please edit the question.

1  
This question is not quite appropriate for MO; you might have better luck at math.stackexchange.com. –  Qiaochu Yuan Feb 25 '11 at 19:52

1 Answer 1

I'm not sure, but it seems to me you're done!: $H_{jk}=\sum_{i=1}^m d_i X_{ij} X_{ik}=\sum_{i=1}^m X^T_{ji} D_{ii} X_{ik}$. Where, as you mention $D_{ii}=d_i$, is a diagonal matrix, therefore: $H_{jk} =\sum_{i=1}^m \sum_{l=1}^m \sum_{n=1}^m X^T_{jl} D_{ln} X_{nk}=X^TDX$.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.