2
$\begingroup$

Suppose we have a typical logdet function $\mathcal{L}$ with respect to a covariance matrix $\mathbf{A}$, $$ \mathcal{L}(\mathbf{A}) = \log\vert \mathbf{I} + \mathbf{A}\mathbf{S} \vert - \mathbf{q}^T(\mathbf{A}^{-1} + \mathbf{S})^{-1} \mathbf{q}, $$ where $\mathbf{S}$ is a Symmetric Positive Semi-Definite Matrix, $\mathbf{q}$ is a column vector, $\mathbf{I}$ is an identity matrix.

The questions are

  1. How could I calculate the gradient of $\mathcal{L}(\mathbf{A})$, Given that $\mathbf{A}$ is a Symmetric Positive Definite Matrix ? I found in literature that if $\mathbf{A}$ is a Symmetric matrix, $\frac{\partial \log\vert \mathbf{A}\vert}{\partial \mathbf{A}}= \mathbf{A}^{-1} + \mathbf{A}^{-T} - \mathbf{I}\odot\mathbf{A}^{-1}$, where $\odot$ is Hadamard Product. However, I found that most of current literatures simply assumes that $\frac{\partial \log\vert \mathbf{A}\vert}{\partial \mathbf{A}} = \mathbf{A}^{-T}$. Will these difference affects the gradient result?

  2. How could I calculate the Hessian of $\mathcal{L}(\mathbf{A})$? i.e., $H = \frac{\partial \mathcal{L}(\mathbf{A})}{\partial^2\mathbf{A}}$, where I need to calculate the derivatives of a Matrix to a Matrix. If I want to find the minima, maxima, saddle points of $\mathbf{H}$, should the result that the Hessian matrix $\mathbf{H}$ being positive definite, negative definite, and none definite still holds ? How could I find the minima by exploiting the Hessian matrix, which is a matrix-by-matrix derivatives.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

I am back with good news: you have nothing to do because there are (in general) no critical points! It is the case if, in particular, $S$ is invertible. Recall that $\mathcal{L}(A)=log|I+AS|-q^T(I+AS)^{-1}Aq$ and (according to my previous post) $D\mathcal{L}_A=\mathcal{L}'(A)=H\rightarrow tr((I+AS)^{-1}HS)-q^T(-(I+AS)^{-1}HS(I+AS)^{-1}A+(I+AS)^{-1}H)q$.

Thus $D\mathcal{L}_A=0$ is equivalent to: for any matrix $H$, $tr(S(I+AS)^{-1}H)=q^T(-(I+AS)^{-1}H(I-S(I+AS)^{-1}A)q$, that is in the form

(*) $tr(UH)=r^THs$ where $r,s$ are vectors and with $U=S(I+AS)^{-1}$.

Lemma: If (*) is true for any $H$, then necessarily $rank(U)\le 1$.

Proof: for any $h_{i,j}$, $\sum_{i,j}u_{j,i}h_{i,j}=\sum_{i,j}r_is_jh_{i,j}$. Thus $u_{j,i}=s_jr_i$, that is $U=sr^T$. Therefore $rank(U)\leq 1$.

Conclusion: Here $A,S\geq 0$; thus $I+AS$ is invertible and consequently, $\mathcal{L}$ admits some critical points only if $rank(S)\leq 1$, that is $S=ww^T$ where $w$ is a vector.

Improve this answer
$\endgroup$
1
$\begingroup$

I assume that $|U|$ denotes $\det(U)$. Let $\phi:U\rightarrow \log|U|$. Then $\phi'(U):H\rightarrow tr(U^{-1}H)$ and $\phi''(U):(H,K) \rightarrow tr(-U^{-1}KU^{-1}H)$. Let '$\mathcal{L_1}(A)=\log|I+AS|$'. Then $\mathcal{L_1}'(A):H \rightarrow tr((I+AS)^{-1}HS)$ and $\mathcal{L_1}''(A):(H,K) \rightarrow tr(-(I+AS)^{-1}KS(I+AS)^{-1}HS)$. Let '$\mathcal{L}_2(A)=(A^{-1}+S)^{-1}=(I+AS)^{-1}A$. Then '$\mathcal{L}_2'(A):H \rightarrow -(I+AS)^{-1}HS(I+AS)^{-1}A+(I+AS)^{-1}H$'. '$\mathcal{L}_2''(A):(H,K) \rightarrow (I+AS)^{-1}KS(I+AS)^{-1}HS(I+AS)^{-1}A+$' $(I+AS)^{-1}HS(I+AS)^{-1}KS(I+AS)^{-1}A-$ $(I+AS)^{-1}HS(I+AS)^{-1}K-(I+AS)^{-1}KS(I+AS)^{-1}H$.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thanks for your reply. But what is $\mathbf{H}$ and $\mathbf{K}$ ? I used to think that the Hessian of $f(\mathbf{A})$ requires vectorized derivatives, i.e., $H = \frac{\partial^2 \mathcal{L}(\mathrm{vec}\mathbf{A})}{\partial \mathrm{vec}(\mathbf{A})\partial \mathrm{vec}^T(\mathbf{A})}$. $\endgroup$
    – liubenyuan
    Jun 21, 2013 at 4:31
  • $\begingroup$ $H,K \in \mathcal{M}_n (\mathbb{R})$. $\phi' (U)$ is a linear function of $H$ and $\phi'' (U)$ is a symmetric bilinear function of $H,K$. $\endgroup$
    – loup blanc
    Jun 21, 2013 at 8:17
  • $\begingroup$ and what is $\mathcal{M}_n(\mathbb{R})$ ? I know that $\frac{\partial \mathcal{L}_1(\mathbf{A})}{\partial \mathbf{A}} = tr((I + AS)^{-1}S))$. I am sorry but I am not so familiar with the notation used in your answer. And if $\frac{\partial \mathcal{L}(\mathbf{A})}{\partial \mathbf{A}} = 0$ is the local minima of $\mathcal{L}(\mathbf{A})$, do I still need to prove the Hessian is p.s.d ? $\endgroup$
    – liubenyuan
    Jun 21, 2013 at 9:14
  • $\begingroup$ $\mathcal{M}_n( \mathbb{R})$ denotes the $n \times n$ matrices with real entries. The derivative of '$\mathcal{L}_1$' cannot be a real. Yet, using the duality associated to the scalar product $(U,V) \rightarrow tr(U^TV)$, you can say that $\phi'(U)={U^{-1}}^T$, and '$\mathcal{L}_1'(A)={(I+AS)^{-1}}^TS^T$'. To solve $\mathcal{L}'(A)=0$ seems to me difficult. If you know explicitly $S,q$, then you can proceed numerically. $\endgroup$
    – loup blanc
    Jun 21, 2013 at 16:43
  • $\begingroup$ Thank you blanc. I never thought the second partial derivative of the matrix could be expressed in a so easy way. BTW, if $\mathbf{S}$ is full ranked square matrix, $\mathcal{L}'(A)=\mathbf{0}$ can be solved in a closed form. $\endgroup$
    – liubenyuan
    Jun 22, 2013 at 5:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.