2
$\begingroup$

Suppose we have a typical logdet function $\mathcal{L}$ with respect to a covariance matrix $\mathbf{A}$, $$ \mathcal{L}(\mathbf{A}) = \log\vert \mathbf{I} + \mathbf{A}\mathbf{S} \vert - \mathbf{q}^T(\mathbf{A}^{-1} + \mathbf{S})^{-1} \mathbf{q}, $$ where $\mathbf{S}$ is a Symmetric Positive Semi-Definite Matrix, $\mathbf{q}$ is a column vector, $\mathbf{I}$ is an identity matrix.

The questions are

  1. How could I calculate the gradient of $\mathcal{L}(\mathbf{A})$, Given that $\mathbf{A}$ is a Symmetric Positive Definite Matrix ? I found in literature that if $\mathbf{A}$ is a Symmetric matrix, $\frac{\partial \log\vert \mathbf{A}\vert}{\partial \mathbf{A}}= \mathbf{A}^{-1} + \mathbf{A}^{-T} - \mathbf{I}\odot\mathbf{A}^{-1}$, where $\odot$ is Hadamard Product. However, I found that most of current literatures simply assumes that $\frac{\partial \log\vert \mathbf{A}\vert}{\partial \mathbf{A}} = \mathbf{A}^{-T}$. Will these difference affects the gradient result?

  2. How could I calculate the Hessian of $\mathcal{L}(\mathbf{A})$? i.e., $H = \frac{\partial \mathcal{L}(\mathbf{A})}{\partial^2\mathbf{A}}$, where I need to calculate the derivatives of a Matrix to a Matrix. If I want to find the minima, maxima, saddle points of $\mathbf{H}$, should the result that the Hessian matrix $\mathbf{H}$ being positive definite, negative definite, and none definite still holds ? How could I find the minima by exploiting the Hessian matrix, which is a matrix-by-matrix derivatives.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

I am back with good news: you have nothing to do because there are (in general) no critical points! It is the case if, in particular, $S$ is invertible. Recall that $\mathcal{L}(A)=log|I+AS|-q^T(I+AS)^{-1}Aq$ and (according to my previous post) $D\mathcal{L}_A=\mathcal{L}'(A)=H\rightarrow tr((I+AS)^{-1}HS)-q^T(-(I+AS)^{-1}HS(I+AS)^{-1}A+(I+AS)^{-1}H)q$.

Thus $D\mathcal{L}_A=0$ is equivalent to: for any matrix $H$, $tr(S(I+AS)^{-1}H)=q^T(-(I+AS)^{-1}H(I-S(I+AS)^{-1}A)q$, that is in the form

(*) $tr(UH)=r^THs$ where $r,s$ are vectors and with $U=S(I+AS)^{-1}$.

Lemma: If (*) is true for any $H$, then necessarily $rank(U)\le 1$.

Proof: for any $h_{i,j}$, $\sum_{i,j}u_{j,i}h_{i,j}=\sum_{i,j}r_is_jh_{i,j}$. Thus $u_{j,i}=s_jr_i$, that is $U=sr^T$. Therefore $rank(U)\leq 1$.

Conclusion: Here $A,S\geq 0$; thus $I+AS$ is invertible and consequently, $\mathcal{L}$ admits some critical points only if $rank(S)\leq 1$, that is $S=ww^T$ where $w$ is a vector.

Improve this answer
$\endgroup$
1
$\begingroup$

I assume that $|U|$ denotes $\det(U)$. Let $\phi:U\rightarrow \log|U|$. Then $\phi'(U):H\rightarrow tr(U^{-1}H)$ and $\phi''(U):(H,K) \rightarrow tr(-U^{-1}KU^{-1}H)$. Let '$\mathcal{L_1}(A)=\log|I+AS|$'. Then $\mathcal{L_1}'(A):H \rightarrow tr((I+AS)^{-1}HS)$ and $\mathcal{L_1}''(A):(H,K) \rightarrow tr(-(I+AS)^{-1}KS(I+AS)^{-1}HS)$. Let '$\mathcal{L}_2(A)=(A^{-1}+S)^{-1}=(I+AS)^{-1}A$. Then '$\mathcal{L}_2'(A):H \rightarrow -(I+AS)^{-1}HS(I+AS)^{-1}A+(I+AS)^{-1}H$'. '$\mathcal{L}_2''(A):(H,K) \rightarrow (I+AS)^{-1}KS(I+AS)^{-1}HS(I+AS)^{-1}A+$' $(I+AS)^{-1}HS(I+AS)^{-1}KS(I+AS)^{-1}A-$ $(I+AS)^{-1}HS(I+AS)^{-1}K-(I+AS)^{-1}KS(I+AS)^{-1}H$.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thanks for your reply. But what is $\mathbf{H}$ and $\mathbf{K}$ ? I used to think that the Hessian of $f(\mathbf{A})$ requires vectorized derivatives, i.e., $H = \frac{\partial^2 \mathcal{L}(\mathrm{vec}\mathbf{A})}{\partial \mathrm{vec}(\mathbf{A})\partial \mathrm{vec}^T(\mathbf{A})}$. $\endgroup$
    – liubenyuan
    Commented Jun 21, 2013 at 4:31
  • $\begingroup$ $H,K \in \mathcal{M}_n (\mathbb{R})$. $\phi' (U)$ is a linear function of $H$ and $\phi'' (U)$ is a symmetric bilinear function of $H,K$. $\endgroup$
    – loup blanc
    Commented Jun 21, 2013 at 8:17
  • $\begingroup$ and what is $\mathcal{M}_n(\mathbb{R})$ ? I know that $\frac{\partial \mathcal{L}_1(\mathbf{A})}{\partial \mathbf{A}} = tr((I + AS)^{-1}S))$. I am sorry but I am not so familiar with the notation used in your answer. And if $\frac{\partial \mathcal{L}(\mathbf{A})}{\partial \mathbf{A}} = 0$ is the local minima of $\mathcal{L}(\mathbf{A})$, do I still need to prove the Hessian is p.s.d ? $\endgroup$
    – liubenyuan
    Commented Jun 21, 2013 at 9:14
  • $\begingroup$ $\mathcal{M}_n( \mathbb{R})$ denotes the $n \times n$ matrices with real entries. The derivative of '$\mathcal{L}_1$' cannot be a real. Yet, using the duality associated to the scalar product $(U,V) \rightarrow tr(U^TV)$, you can say that $\phi'(U)={U^{-1}}^T$, and '$\mathcal{L}_1'(A)={(I+AS)^{-1}}^TS^T$'. To solve $\mathcal{L}'(A)=0$ seems to me difficult. If you know explicitly $S,q$, then you can proceed numerically. $\endgroup$
    – loup blanc
    Commented Jun 21, 2013 at 16:43
  • $\begingroup$ Thank you blanc. I never thought the second partial derivative of the matrix could be expressed in a so easy way. BTW, if $\mathbf{S}$ is full ranked square matrix, $\mathcal{L}'(A)=\mathbf{0}$ can be solved in a closed form. $\endgroup$
    – liubenyuan
    Commented Jun 22, 2013 at 5:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .