# Requirements for finite backward derivatives of degenerate eigenvectors

A matrix, $$\mathbf{A}(\theta)\in\mathbb{R}^{n\times n}$$, has elements that depend on a parameter $$\theta$$. The $$j$$-th eigenvalues and eigenvectors of the matrix are denoted as $$\lambda_j$$ and $$\mathbf{x}_j$$, respectively. I would like to know the requirements to obtain finite backward derivatives of eigenvectors in degenerate case.

Let me elaborate. The forward derivative of the eigendecomposition is given by $$\delta\mathbf{x}_j = -\sum_{i\neq j} (\lambda_i - \lambda_j)^{-1} \mathbf{x}_i \left[\mathbf{x}_i^T (\delta \mathbf{A}) \mathbf{x}_j\right],$$ where $$\delta\mathbf{A}$$ and $$\delta \mathbf{x}_j$$ are small changes of each element in $$\mathbf{A}$$ and $$\mathbf{x}_j$$, respectively.

If there is a degeneracy, $$\lambda_d = \lambda_j\ \forall\ d\in\mathrm{degen}(j)$$, where $$\mathrm{degen}(j)$$ is the set of degenerate indices with the same eigenvalues of $$\lambda_j$$, excluding $$j$$. The requirement to get finite $$\delta \mathbf{x}_j$$ is $$\mathbf{x}_d^T(\delta\mathbf{A})\mathbf{x}_j = 0\ \forall\ d\in \mathrm{degen}(j).$$

The backward derivative, on the other hand, is given by $$\frac{\partial \mathcal{L}}{\partial \mathbf{A}} = -\sum_j\sum_{i\neq j}(\lambda_i - \lambda_j)^{-1}\mathbf{x}_i \mathbf{x}_i^T \frac{\partial \mathcal{L}}{\partial \mathbf{x}_j} \mathbf{x}_j^T,$$ where $$\mathcal{L}$$ is a loss value, $$\frac{\partial \mathcal{L}}{\partial \mathbf{x}_j}\in\mathbb{R}^{n\times 1}$$ and $$\frac{\partial \mathcal{L}}{\partial \mathbf{A}}\in\mathbb{R}^{n\times n}$$ are the change in the loss value w.r.t. each element in $$\mathbf{x}_j$$ and $$\mathbf{A}$$, respectively.

In case of degeneracy, $$\mathrm{degen}(j)\neq \emptyset$$, is it possible to get finite $$\frac{\partial \mathcal{L}}{\partial \mathbf{A}}$$? If so, what are the requirements? If we don't care about $$\frac{\partial \mathcal{L}}{\partial \mathbf{A}}$$, can we relax the requirements just to get finite $$\frac{\partial \mathcal{L}}{\partial \theta}$$ ($$\theta$$ is the parameter the elements of $$\mathbf{A}$$ depends on)?

Let's take $$m \in \mathrm{degen}(n)$$. The terms involving $$m$$ and $$n$$ in $$\frac{\partial \mathcal{L}}{\partial \mathbf{A}}$$ are $$\frac{\partial\mathcal{L}}{\partial \mathbf{A}} =-(\lambda_m -\lambda_n)^{-1}\left[\mathbf{x}_m\mathbf{x}_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_n}\mathbf{x}_n^T - \mathbf{x}_n\mathbf{x}_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_m}\mathbf{x}_m^T\right] + \ ...$$ Therefore, to get finite $$\frac{\partial\mathcal{L}}{\partial \mathbf{A}}$$, one needs $$\mathbf{x}_m\mathbf{x}_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_n}\mathbf{x}_n^T = \mathbf{x}_n\mathbf{x}_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_m}\mathbf{x}_m^T\ \forall\ m\in\mathrm{degen}(n)$$
If we don't care about $$\frac{\partial\mathcal{L}}{\partial \mathbf{A}}$$, only $$\frac{\partial\mathcal{L}}{\partial \theta}$$, we should write the expression for $$\frac{\partial\mathcal{L}}{\partial \theta}$$ first,
$$\frac{\partial\mathcal{L}}{\partial \theta} = \mathrm{tr}\left[\left(\frac{\partial\mathcal{L}}{\partial \mathbf{A}}\right)^T\frac{\partial \mathbf{A}}{\partial \theta}\right]$$
The term in $$\frac{\partial\mathcal{L}}{\partial \theta}$$ involving $$m$$ and $$n$$ are $$\frac{\partial\mathcal{L}}{\partial \theta} = \mathrm{tr}\left[(\lambda_m-\lambda_n)^{-1}\left(\mathbf{x}_m\mathbf{x}_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_n}\mathbf{x}_n^T - \mathbf{x}_n\mathbf{x}_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_m}\mathbf{x}_m^T\right)^T\frac{\partial \mathbf{A}}{\partial \theta}\right] + \ ...$$ Therefore, to get finite $$\frac{\partial \mathcal{L}}{\partial \theta}$$, the following condition must be satisfied \begin{align} \mathrm{tr}\left[\left(\mathbf{x}_m\mathbf{x}_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_n}\mathbf{x}_n^T\right)^T\frac{\partial \mathbf{A}}{\partial \theta}\right] &= \mathrm{tr}\left[\left(\mathbf{x}_n\mathbf{x}_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_m}\mathbf{x}_m^T\right)^T\frac{\partial \mathbf{A}}{\partial \theta}\right] \\ \mathrm{tr}\left[\left(\mathbf{x}_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_n}\right)\mathbf{x}_m^T\frac{\partial \mathbf{A}}{\partial \theta}\mathbf{x}_n\right] &= \mathrm{tr}\left[\left(\mathbf{x}_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_m}\right)\mathbf{x}_n^T\frac{\partial \mathbf{A}}{\partial \theta}\mathbf{x}_m\right] \\ \left(\mathbf{x}_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_n}\right)\mathbf{x}_m^T\frac{\partial \mathbf{A}}{\partial \theta}\mathbf{x}_n &= \left(\mathbf{x}_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_m}\right)\mathbf{x}_n^T\frac{\partial \mathbf{A}}{\partial \theta}\mathbf{x}_m \end{align}
If the matrix $$\mathbf{A}$$ is always symmetric for all values of $$\theta$$, i.e. $$\frac{\partial\mathbf{A}}{\partial \theta}$$ is always symmetric, then there are two conditions that can satisfy the condition above:
\begin{align} \mathbf{x}_n^T\frac{\partial \mathbf{A}}{\partial \theta}\mathbf{x}_m &= 0\\ & \mathrm{or} \\ \mathbf{x}_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_n} &= \mathbf{x}_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_m} \end{align}