Regularized Gradient with respect to a matrix (with a specific structure)

Question

Suppose we have a typical logdet function $\mathcal{L}$ $$ \mathcal{L} = \log\vert \mathbf{I} + \mathbf{A}\mathbf{S} \vert - \mathbf{q}^T(\mathbf{A}^{-1} + \mathbf{S})^{-1} \mathbf{q}, $$ where $\mathbf{S}$ is a Symmetric Positive Semi-Definite Matrix, $\mathbf{q}$ is a column vector, $\mathbf{I}$ is an identity matrix. The local minima of $\mathcal{L}$ w.r.t $\mathbf{A}$ could be found by solving $\mathbf{A}$ ($\frac{\partial \mathcal{L}}{\partial \mathbf{A}} = \mathbf{0}$), we get the closed form solution of $\mathbf{A}$ $$ \mathbf{A} = \mathbf{S}^{-1}(\mathbf{S} - \mathbf{q}\mathbf{q}^T)\mathbf{S}^{-1} $$

The problem is, what if I want to constrain the descent path of the logdet function by imposing extra structures on $\mathbf{A}$ ? For example,

1. $\mathbf{A}$ is p.s.d and have a simple structure, which constrains that at each step, $\mathbf{A}$ should be: $\mathbf{A}=r\mathbf{I}$.

2. or we constrain that $\mathbf{A}$ is p.s.d and have a band-p Symmetric Toeplitz structure, i.e., \begin{equation} \mathbf{A} = r \begin{pmatrix} 1 & p & 0 & 0 \\\ p & 1 & p & 0 \\\ 0 & p & 1 & p \\\ 0 & 0 & p & 1 \end{pmatrix}. \end{equation}

How could I analyze the saddle point of $\mathcal{L}$ with these constrains ? Will these constrains affect achieving the local minima of $\mathcal{L}$ and provide a more robust result that more likely to find the true minimal that has this type of structure ?

Answer 1

@liubenyuan, some remarks. Necessarily $S>0$. On the other hand, your result is correct up a signum (have a look at the case $n=1$); infortunately that makes a big difference ! The derivative is $0$ if $A=S^{-1}(qq^T-S)S^{-1}$ and you want that $A\geq 0$, that is $qq^T\geq S$. Let $(\lambda_i\geq 0)_i$ be the spectrum of $S$. According to the min-max theorem $||q||^2\geq \lambda_1$ and, for every $i>1$, $\lambda_i=0$ and $S$ cannot be invertible except if $n=1$. Finally, there are no critical points, except for $n=1$, $a=q^2/s^2-1/s$ with $q^2\geq s$.

EDIT: cf. the following post. Have you read my answer to this post ?

Hessian of function of covariance matrices