Given a large real symmetric matrix, $\mathbf{A(m)}\in\mathbb{R}^{N\times N}$, which elements depend on a parameters vector, $\mathbf{m}\in\mathbb{R}^m$. The matrix elements cannot be stored explicitly, but we can compute the matrix-vector operation, $\mathbf{A(m)v}$.

Let's say we have obtained $B$ ($B\ll N$) lowest eigenvectors and eigenvalues of $\mathbf{A(m)}$ as $\mathbf{V_B}$ and $\mathbf{\Lambda_B}$, respectively. The matrix of eigenvectors and diagonal matrix of eigenvalues satisfy $$ \mathbf{A(m)V_B = V_B\Lambda_B}. $$ Is there any iterative or direct method to calculate the derivative of a defined loss function with respect to every elements of $\mathbf{m}$, i.e. $\partial \mathcal{L}/\partial \mathbf{m}$ given $\partial \mathcal{L}/\partial \mathbf{V_B}$, $\partial \mathcal{L}/\partial \mathbf{\Lambda_B}$, $\mathbf{V_B}$, $\mathbf{\Lambda_B}$? The operation $\frac{\partial \mathbf{A}}{\partial m_i}\mathbf{v}$ can also be computed.

Please note that the matrix $\mathbf{A}$ is too large to be stored in memory, so I want to avoid storing all the eigenvectors (but storing $\mathbf{V_B}$ is fine).

For a case where $\partial \mathcal{L}/\partial \mathbf{V_B} = \mathbf{0}$, I can solve the problem. From perturbation theory, we perturbation of the $b$-th eigenvalue, $\lambda_b$, is given by $$ \delta\lambda_b = \mathbf{v}^T_\mathbf{b}\mathbf{\delta A}\mathbf{v_b}. $$ I assume $\mathbf{\delta A}$ is symmetric to preserve the symmetricity of the matrix $\mathbf{A}$. Then, the derivative of a loss function with respect to every element in $\mathbf{A}$ is given by $$ \frac{\partial \mathcal{L}}{\partial \mathbf{A}}=\sum_{b}\mathbf{v_b}\mathbf{v}_{\mathbf{b}}^T \frac{\partial \mathcal{L}}{\partial \lambda_b} = \mathbf{V_B}\frac{\partial \mathcal{L}}{\partial \mathbf{\Lambda_B}}\mathbf{V}^T_\mathbf{B}.$$

Using the chain rule, we obtain $$ \frac{\partial \mathcal{L}}{\partial m_i} = \sum_{j,k}\frac{\partial\mathcal{L}}{\partial \mathbf{A}_{jk}} \frac{\partial \mathbf{A}_{jk}}{\partial m_i}=\mathrm{tr}\left[\mathbf{V}^T_\mathbf{B}\frac{\partial \mathcal{L}}{\partial \mathbf{\Lambda_B}}\frac{\partial\mathbf{A}}{\partial m_i}\mathbf{V_B}\right] $$

The problem is, if $\partial \mathcal{L}/\partial \mathbf{V_B} \ne \mathbf{0}$, I cannot find the expression that does not involve the full matrix $\mathbf{A}$ or the full eigenvalues matrix.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.