Requirements for finite backward derivatives of degenerate eigenvectors A matrix, $\mathbf{A}(\theta)\in\mathbb{R}^{n\times n}$, has elements that depend on a parameter $\theta$. The $j$-th eigenvalues and eigenvectors of the matrix are denoted as $\lambda_j$ and $\mathbf{x}_j$, respectively.
I would like to know the requirements to obtain finite backward derivatives of eigenvectors in degenerate case.
Let me elaborate.
The forward derivative of the eigendecomposition is given by
$$
\delta\mathbf{x}_j = -\sum_{i\neq j} (\lambda_i - \lambda_j)^{-1} \mathbf{x}_i \left[\mathbf{x}_i^T (\delta \mathbf{A}) \mathbf{x}_j\right],
$$
where $\delta\mathbf{A}$ and $\delta \mathbf{x}_j$ are small changes of each element in $\mathbf{A}$ and $\mathbf{x}_j$, respectively.
If there is a degeneracy, $\lambda_d = \lambda_j\ \forall\ d\in\mathrm{degen}(j)$, where $\mathrm{degen}(j)$ is the set of degenerate indices with the same eigenvalues of $\lambda_j$, excluding $j$.
The requirement to get finite $\delta \mathbf{x}_j$ is
$$
\mathbf{x}_d^T(\delta\mathbf{A})\mathbf{x}_j = 0\ \forall\ d\in \mathrm{degen}(j).
$$
The backward derivative, on the other hand, is given by
$$
\frac{\partial \mathcal{L}}{\partial \mathbf{A}} = -\sum_j\sum_{i\neq j}(\lambda_i - \lambda_j)^{-1}\mathbf{x}_i \mathbf{x}_i^T \frac{\partial \mathcal{L}}{\partial \mathbf{x}_j} \mathbf{x}_j^T,
$$
where $\mathcal{L}$ is a loss value, $\frac{\partial \mathcal{L}}{\partial \mathbf{x}_j}\in\mathbb{R}^{n\times 1}$ and $\frac{\partial \mathcal{L}}{\partial \mathbf{A}}\in\mathbb{R}^{n\times n}$ are the  change in the loss value w.r.t. each element in $\mathbf{x}_j$ and $\mathbf{A}$, respectively.
In case of degeneracy, $\mathrm{degen}(j)\neq \emptyset$, is it possible to get finite $\frac{\partial \mathcal{L}}{\partial \mathbf{A}}$? If so, what are the requirements?
If we don't care about $\frac{\partial \mathcal{L}}{\partial \mathbf{A}}$, can we relax the requirements just to get finite $\frac{\partial \mathcal{L}}{\partial \theta}$ ($\theta$ is the parameter the elements of $\mathbf{A}$ depends on)?
 A: Let's take $m \in \mathrm{degen}(n)$. The terms involving $m$ and $n$ in $\frac{\partial \mathcal{L}}{\partial \mathbf{A}}$ are
$$
\frac{\partial\mathcal{L}}{\partial \mathbf{A}} =-(\lambda_m -\lambda_n)^{-1}\left[\mathbf{x}_m\mathbf{x}_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_n}\mathbf{x}_n^T - \mathbf{x}_n\mathbf{x}_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_m}\mathbf{x}_m^T\right] + \ ...
$$
Therefore, to get finite $\frac{\partial\mathcal{L}}{\partial \mathbf{A}}$, one needs
$$
\mathbf{x}_m\mathbf{x}_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_n}\mathbf{x}_n^T = \mathbf{x}_n\mathbf{x}_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_m}\mathbf{x}_m^T\ \forall\ m\in\mathrm{degen}(n)
$$
If we don't care about $\frac{\partial\mathcal{L}}{\partial \mathbf{A}}$, only $\frac{\partial\mathcal{L}}{\partial \theta}$, we should write the expression for $\frac{\partial\mathcal{L}}{\partial \theta}$ first,
$$
\frac{\partial\mathcal{L}}{\partial \theta} = \mathrm{tr}\left[\left(\frac{\partial\mathcal{L}}{\partial \mathbf{A}}\right)^T\frac{\partial \mathbf{A}}{\partial \theta}\right]
$$
The term in $\frac{\partial\mathcal{L}}{\partial \theta}$ involving $m$ and $n$ are
$$
\frac{\partial\mathcal{L}}{\partial \theta} = \mathrm{tr}\left[(\lambda_m-\lambda_n)^{-1}\left(\mathbf{x}_m\mathbf{x}_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_n}\mathbf{x}_n^T - \mathbf{x}_n\mathbf{x}_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_m}\mathbf{x}_m^T\right)^T\frac{\partial \mathbf{A}}{\partial \theta}\right] + \ ...
$$
Therefore, to get finite $\frac{\partial \mathcal{L}}{\partial \theta}$, the following condition must be satisfied
$$
\begin{align}
\mathrm{tr}\left[\left(\mathbf{x}_m\mathbf{x}_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_n}\mathbf{x}_n^T\right)^T\frac{\partial \mathbf{A}}{\partial \theta}\right] &= 
\mathrm{tr}\left[\left(\mathbf{x}_n\mathbf{x}_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_m}\mathbf{x}_m^T\right)^T\frac{\partial \mathbf{A}}{\partial \theta}\right] \\
\mathrm{tr}\left[\left(\mathbf{x}_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_n}\right)\mathbf{x}_m^T\frac{\partial \mathbf{A}}{\partial \theta}\mathbf{x}_n\right] &= \mathrm{tr}\left[\left(\mathbf{x}_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_m}\right)\mathbf{x}_n^T\frac{\partial \mathbf{A}}{\partial \theta}\mathbf{x}_m\right] \\
\left(\mathbf{x}_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_n}\right)\mathbf{x}_m^T\frac{\partial \mathbf{A}}{\partial \theta}\mathbf{x}_n &= \left(\mathbf{x}_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_m}\right)\mathbf{x}_n^T\frac{\partial \mathbf{A}}{\partial \theta}\mathbf{x}_m
\end{align}
$$
If the matrix $\mathbf{A}$ is always symmetric for all values of $\theta$, i.e. $\frac{\partial\mathbf{A}}{\partial \theta}$ is always symmetric, then there are two conditions that can satisfy the condition above:
$$
\begin{align}
\mathbf{x}_n^T\frac{\partial \mathbf{A}}{\partial \theta}\mathbf{x}_m &= 0\\
& \mathrm{or} \\
\mathbf{x}_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_n} &= \mathbf{x}_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}_m}
\end{align}
$$
