1
$\begingroup$

A matrix, $\mathbf{A}(\theta)\in\mathbb{R}^{n\times n}$, has elements that depend on a parameter $\theta$. The $j$-th eigenvalues and eigenvectors of the matrix are denoted as $\lambda_j$ and $\mathbf{x}_j$, respectively. I would like to know the requirements to obtain finite backward derivatives of eigenvectors in degenerate case.

Let me elaborate. The forward derivative of the eigendecomposition is given by $$ \delta\mathbf{x}_j = -\sum_{i\neq j} (\lambda_i - \lambda_j)^{-1} \mathbf{x}_i \left[\mathbf{x}_i^T (\delta \mathbf{A}) \mathbf{x}_j\right], $$ where $\delta\mathbf{A}$ and $\delta \mathbf{x}_j$ are small changes of each element in $\mathbf{A}$ and $\mathbf{x}_j$, respectively.

If there is a degeneracy, $\lambda_d = \lambda_j\ \forall\ d\in\mathrm{degen}(j)$, where $\mathrm{degen}(j)$ is the set of degenerate indices with the same eigenvalues of $\lambda_j$, excluding $j$. The requirement to get finite $\delta \mathbf{x}_j$ is $$ \mathbf{x}_d^T(\delta\mathbf{A})\mathbf{x}_j = 0\ \forall\ d\in \mathrm{degen}(j). $$

The backward derivative, on the other hand, is given by $$ \frac{\partial \mathcal{L}}{\partial \mathbf{A}} = -\sum_j\sum_{i\neq j}(\lambda_i - \lambda_j)^{-1}\mathbf{x}_i \mathbf{x}_i^T \frac{\partial \mathcal{L}}{\partial \mathbf{x}_j} \mathbf{x}_j^T, $$ where $\mathcal{L}$ is a loss value, $\frac{\partial \mathcal{L}}{\partial \mathbf{x}_j}\in\mathbb{R}^{n\times 1}$ and $\frac{\partial \mathcal{L}}{\partial \mathbf{A}}\in\mathbb{R}^{n\times n}$ are the change in the loss value w.r.t. each element in $\mathbf{x}_j$ and $\mathbf{A}$, respectively.

In case of degeneracy, $\mathrm{degen}(j)\neq \emptyset$, is it possible to get finite $\frac{\partial \mathcal{L}}{\partial \mathbf{A}}$? If so, what are the requirements? If we don't care about $\frac{\partial \mathcal{L}}{\partial \mathbf{A}}$, can we relax the requirements just to get finite $\frac{\partial \mathcal{L}}{\partial \theta}$ ($\theta$ is the parameter the elements of $\mathbf{A}$ depends on)?

$\endgroup$
0
$\begingroup$

Let's take $m \in \mathrm{degen}(n)$. The terms involving $m$ and $n$ in $\frac{\partial \mathcal{L}}{\partial \mathbf{A}}$ are $$ \frac{\partial\mathcal{L}}{\partial \mathbf{A}} =-(\lambda_m -\lambda_n)^{-1}\left[\mathbf{x}_m\mathbf{x}_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_n}\mathbf{x}_n^T - \mathbf{x}_n\mathbf{x}_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_m}\mathbf{x}_m^T\right] + \ ... $$ Therefore, to get finite $\frac{\partial\mathcal{L}}{\partial \mathbf{A}}$, one needs $$ \mathbf{x}_m\mathbf{x}_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_n}\mathbf{x}_n^T = \mathbf{x}_n\mathbf{x}_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_m}\mathbf{x}_m^T\ \forall\ m\in\mathrm{degen}(n) $$

If we don't care about $\frac{\partial\mathcal{L}}{\partial \mathbf{A}}$, only $\frac{\partial\mathcal{L}}{\partial \theta}$, we should write the expression for $\frac{\partial\mathcal{L}}{\partial \theta}$ first,

$$ \frac{\partial\mathcal{L}}{\partial \theta} = \mathrm{tr}\left[\left(\frac{\partial\mathcal{L}}{\partial \mathbf{A}}\right)^T\frac{\partial \mathbf{A}}{\partial \theta}\right] $$

The term in $\frac{\partial\mathcal{L}}{\partial \theta}$ involving $m$ and $n$ are $$ \frac{\partial\mathcal{L}}{\partial \theta} = \mathrm{tr}\left[(\lambda_m-\lambda_n)^{-1}\left(\mathbf{x}_m\mathbf{x}_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_n}\mathbf{x}_n^T - \mathbf{x}_n\mathbf{x}_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_m}\mathbf{x}_m^T\right)^T\frac{\partial \mathbf{A}}{\partial \theta}\right] + \ ... $$ Therefore, to get finite $\frac{\partial \mathcal{L}}{\partial \theta}$, the following condition must be satisfied $$ \begin{align} \mathrm{tr}\left[\left(\mathbf{x}_m\mathbf{x}_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_n}\mathbf{x}_n^T\right)^T\frac{\partial \mathbf{A}}{\partial \theta}\right] &= \mathrm{tr}\left[\left(\mathbf{x}_n\mathbf{x}_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_m}\mathbf{x}_m^T\right)^T\frac{\partial \mathbf{A}}{\partial \theta}\right] \\ \mathrm{tr}\left[\left(\mathbf{x}_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_n}\right)\mathbf{x}_m^T\frac{\partial \mathbf{A}}{\partial \theta}\mathbf{x}_n\right] &= \mathrm{tr}\left[\left(\mathbf{x}_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_m}\right)\mathbf{x}_n^T\frac{\partial \mathbf{A}}{\partial \theta}\mathbf{x}_m\right] \\ \left(\mathbf{x}_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_n}\right)\mathbf{x}_m^T\frac{\partial \mathbf{A}}{\partial \theta}\mathbf{x}_n &= \left(\mathbf{x}_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_m}\right)\mathbf{x}_n^T\frac{\partial \mathbf{A}}{\partial \theta}\mathbf{x}_m \end{align} $$

If the matrix $\mathbf{A}$ is always symmetric for all values of $\theta$, i.e. $\frac{\partial\mathbf{A}}{\partial \theta}$ is always symmetric, then there are two conditions that can satisfy the condition above:

$$ \begin{align} \mathbf{x}_n^T\frac{\partial \mathbf{A}}{\partial \theta}\mathbf{x}_m &= 0\\ & \mathrm{or} \\ \mathbf{x}_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_n} &= \mathbf{x}_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_m} \end{align} $$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.