Welcome to MathOverflow

Question

3

I'm following the open courseware content on Machine Learning from Stanford University. In the lecture notes, it is given that

$$\Delta_A \ tr(ABA^TC) = CAB + C^TAB^T$$

which I tried but couldn't prove easily. It is not required to follow the course content but I just wondered and wanted to learn its proof. Any suggestions?

Update: $A$, $B$, and $C$ are matrices and $\Delta_A$ is the gradient operation on matrix $A$.

edited Apr 11 2011 at 8:35

asked Feb 27 2011 at 16:40

İsmail Arı
161●9

2

Dear Ismail: it's not necessary to open with a salutation or have a closing like "kind wishes". I guess $A$, $B$, $C$ are square matrices of the same size; could you say what is meant by $\Delta_A$ please? Also, a direct link to the lecture notes would be very nice. – Todd Trimble Feb 27 2011 at 16:56

1

Dear Todd: Thanks for you advice. I will take your remarks about how to ask a question into consideration. The course notes are here: stanford.edu/class/cs229/notes/cs229-notes1.pdf Page 9, Eq. 3. – İsmail Arı Feb 27 2011 at 17:07

2

I don't see the reason for getting a -1 for the question. Is that because I couldn't prove it easily? If so, why are there 8 current upvotes for the answer post. If I know the reason, I would update my questioning to mathoverflow community accordingly. – İsmail Arı Apr 10 2011 at 23:23

1

I didn't downvote the question, but I can see a good reason for doing so, namely, the carelessness with which the question was presented. No indication of what $A$, $B$, $C$, and $\Delta_A$ were, not editing the question to include this information when the defect was brought to your attention - in fact, I've just decided to downvote the question. – Gerry Myerson Apr 11 2011 at 2:18

Thanks for the warning. I updated the question. – İsmail Arı Apr 11 2011 at 8:27

Fabian · Accepted Answer · 2011-02-27 17:05:58Z UTC

I guess $\Delta_A$ denotes the derivative with respect to the elements of the matrix $A$ (more conventionally denoted by $\partial_{A}$).

To evaluate the derivative with respect to $A_{ij}$, write out the trace in terms of components and then use $\partial_{A_{ij}} A_{mn} = \delta_{im} \delta_{jn}$, $$\partial_{A_{ij}} \text{tr}(A B A^T C) = \partial_{A_{ij}}\sum_{mnkl} A_{mn} B_{nk} A_{lk} C_{lm}= \sum_{kl} B_{jk}A_{lk} C_{li} + \sum_{mn} A_{mn} B_{nj} C_{im} $$ $$= ( C^T A B^T+ C A B )_{ij}.$$ This is the component-wise version of your identity.

Note to the comment of Todd Trimble: the matrices $A,B$, and $C$ do not have to be necessarily square matrices. Their dimension just has to "match" ($A \in \mathbb{R}^{m \times n}$, $B\in \mathbb{R}^{n \times n}$, $C \in \mathbb{R}^{m \times m}$, with $m$ and $n$ arbitrary integers).

Proof of a fact about traces

Remember to vote up questions/answers you find interesting or helpful (requires 15 reputation points)

1 Answer

You can accept an answer to one of your own questions by clicking the check mark next to it. This awards 15 reputation points to the person who answered and 2 reputation points to you.