Take the 2-minute tour ×
MathOverflow is a question and answer site for professional mathematicians. It's 100% free, no registration required.

Suppose we have a typical logdet function $\mathcal{L}$ with respect to a covariance matrix $\mathbf{A}$, $$ \mathcal{L}(\mathbf{A}) = \log\vert \mathbf{I} + \mathbf{A}\mathbf{S} \vert - \mathbf{q}^T(\mathbf{A}^{-1} + \mathbf{S})^{-1} \mathbf{q}, $$ where $\mathbf{S}$ is a Symmetric Positive Semi-Definite Matrix, $\mathbf{q}$ is a column vector, $\mathbf{I}$ is an identity matrix.

The questions are

  1. How could I calculate the gradient of $\mathcal{L}(\mathbf{A})$, Given that $\mathbf{A}$ is a Symmetric Positive Definite Matrix ? I found in literature that if $\mathbf{A}$ is a Symmetric matrix, $\frac{\partial \log\vert \mathbf{A}\vert}{\partial \mathbf{A}}= \mathbf{A}^{-1} + \mathbf{A}^{-T} - \mathbf{I}\odot\mathbf{A}^{-1}$, where $\odot$ is Hadamard Product. However, I found that most of current literatures simply assumes that $\frac{\partial \log\vert \mathbf{A}\vert}{\partial \mathbf{A}} = \mathbf{A}^{-T}$. Will these difference affects the gradient result?

  2. How could I calculate the Hessian of $\mathcal{L}(\mathbf{A})$? i.e., $H = \frac{\partial \mathcal{L}(\mathbf{A})}{\partial^2\mathbf{A}}$, where I need to calculate the derivatives of a Matrix to a Matrix. If I want to find the minima, maxima, saddle points of $\mathbf{H}$, should the result that the Hessian matrix $\mathbf{H}$ being positive definite, negative definite, and none definite still holds ? How could I find the minima by exploiting the Hessian matrix, which is a matrix-by-matrix derivatives.

share|improve this question
add comment

2 Answers

I am back with good news: you have nothing to do because there are (in general) no critical points! It is the case if, in particular, $S$ is invertible. Recall that $\mathcal{L}(A)=log|I+AS|-q^T(I+AS)^{-1}Aq$ and (according to my previous post) $D\mathcal{L}_A=\mathcal{L}'(A)=H\rightarrow tr((I+AS)^{-1}HS)-q^T(-(I+AS)^{-1}HS(I+AS)^{-1}A+(I+AS)^{-1}H)q$.

Thus $D\mathcal{L}_A=0$ is equivalent to: for any matrix $H$, $tr(S(I+AS)^{-1}H)=q^T(-(I+AS)^{-1}H(I-S(I+AS)^{-1}A)q$, that is in the form

(*) $tr(UH)=r^THs$ where $r,s$ are vectors and with $U=S(I+AS)^{-1}$.

Lemma: If (*) is true for any $H$, then necessarily $rank(U)\le 1$.

Proof: for any $h_{i,j}$, $\sum_{i,j}u_{j,i}h_{i,j}=\sum_{i,j}r_is_jh_{i,j}$. Thus $u_{j,i}=s_jr_i$, that is $U=sr^T$. Therefore $rank(U)\leq 1$.

Conclusion: Here $A,S\geq 0$; thus $I+AS$ is invertible and consequently, $\mathcal{L}$ admits some critical points only if $rank(S)\leq 1$, that is $S=ww^T$ where $w$ is a vector.

share|improve this answer
add comment

I assume that $|U|$ denotes $\det(U)$. Let $\phi:U\rightarrow \log|U|$. Then $\phi'(U):H\rightarrow tr(U^{-1}H)$ and $\phi''(U):(H,K) \rightarrow tr(-U^{-1}KU^{-1}H)$. Let '$\mathcal{L_1}(A)=\log|I+AS|$'. Then $\mathcal{L_1}'(A):H \rightarrow tr((I+AS)^{-1}HS)$ and $\mathcal{L_1}''(A):(H,K) \rightarrow tr(-(I+AS)^{-1}KS(I+AS)^{-1}HS)$. Let '$\mathcal{L}_2(A)=(A^{-1}+S)^{-1}=(I+AS)^{-1}A$. Then '$\mathcal{L}_2'(A):H \rightarrow -(I+AS)^{-1}HS(I+AS)^{-1}A+(I+AS)^{-1}H$'. '$\mathcal{L}_2''(A):(H,K) \rightarrow (I+AS)^{-1}KS(I+AS)^{-1}HS(I+AS)^{-1}A+$' $(I+AS)^{-1}HS(I+AS)^{-1}KS(I+AS)^{-1}A-$ $(I+AS)^{-1}HS(I+AS)^{-1}K-(I+AS)^{-1}KS(I+AS)^{-1}H$.

share|improve this answer
    
Thanks for your reply. But what is $\mathbf{H}$ and $\mathbf{K}$ ? I used to think that the Hessian of $f(\mathbf{A})$ requires vectorized derivatives, i.e., $H = \frac{\partial^2 \mathcal{L}(\mathrm{vec}\mathbf{A})}{\partial \mathrm{vec}(\mathbf{A})\partial \mathrm{vec}^T(\mathbf{A})}$. –  liubenyuan Jun 21 '13 at 4:31
    
$H,K \in \mathcal{M}_n (\mathbb{R})$. $\phi' (U)$ is a linear function of $H$ and $\phi'' (U)$ is a symmetric bilinear function of $H,K$. –  loup blanc Jun 21 '13 at 8:17
    
and what is $\mathcal{M}_n(\mathbb{R})$ ? I know that $\frac{\partial \mathcal{L}_1(\mathbf{A})}{\partial \mathbf{A}} = tr((I + AS)^{-1}S))$. I am sorry but I am not so familiar with the notation used in your answer. And if $\frac{\partial \mathcal{L}(\mathbf{A})}{\partial \mathbf{A}} = 0$ is the local minima of $\mathcal{L}(\mathbf{A})$, do I still need to prove the Hessian is p.s.d ? –  liubenyuan Jun 21 '13 at 9:14
    
$\mathcal{M}_n( \mathbb{R})$ denotes the $n \times n$ matrices with real entries. The derivative of '$\mathcal{L}_1$' cannot be a real. Yet, using the duality associated to the scalar product $(U,V) \rightarrow tr(U^TV)$, you can say that $\phi'(U)={U^{-1}}^T$, and '$\mathcal{L}_1'(A)={(I+AS)^{-1}}^TS^T$'. To solve $\mathcal{L}'(A)=0$ seems to me difficult. If you know explicitly $S,q$, then you can proceed numerically. –  loup blanc Jun 21 '13 at 16:43
    
Thank you blanc. I never thought the second partial derivative of the matrix could be expressed in a so easy way. BTW, if $\mathbf{S}$ is full ranked square matrix, $\mathcal{L}'(A)=\mathbf{0}$ can be solved in a closed form. –  liubenyuan Jun 22 '13 at 5:20
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.