0
$\begingroup$

Given a large real symmetric matrix, $\mathbf{A(m)}\in\mathbb{R}^{N\times N}$, which elements depend on a parameters vector, $\mathbf{m}\in\mathbb{R}^m$. The matrix elements cannot be stored explicitly, but we can compute the matrix-vector operation, $\mathbf{A(m)v}$.

Let's say we have obtained $B$ ($B\ll N$) lowest eigenvectors and eigenvalues of $\mathbf{A(m)}$ as $\mathbf{V_B}$ and $\mathbf{\Lambda_B}$, respectively. The matrix of eigenvectors and diagonal matrix of eigenvalues satisfy $$ \mathbf{A(m)V_B = V_B\Lambda_B}. $$ Is there any iterative or direct method to calculate the derivative of a defined loss function with respect to every elements of $\mathbf{m}$, i.e. $\partial \mathcal{L}/\partial \mathbf{m}$ given $\partial \mathcal{L}/\partial \mathbf{V_B}$, $\partial \mathcal{L}/\partial \mathbf{\Lambda_B}$, $\mathbf{V_B}$, $\mathbf{\Lambda_B}$? The operation $\frac{\partial \mathbf{A}}{\partial m_i}\mathbf{v}$ can also be computed.

Please note that the matrix $\mathbf{A}$ is too large to be stored in memory, so I want to avoid storing all the eigenvectors (but storing $\mathbf{V_B}$ is fine).


For a case where $\partial \mathcal{L}/\partial \mathbf{V_B} = \mathbf{0}$, I can solve the problem. From perturbation theory, we perturbation of the $b$-th eigenvalue, $\lambda_b$, is given by $$ \delta\lambda_b = \mathbf{v}^T_\mathbf{b}\mathbf{\delta A}\mathbf{v_b}. $$ I assume $\mathbf{\delta A}$ is symmetric to preserve the symmetricity of the matrix $\mathbf{A}$. Then, the derivative of a loss function with respect to every element in $\mathbf{A}$ is given by $$ \frac{\partial \mathcal{L}}{\partial \mathbf{A}}=\sum_{b}\mathbf{v_b}\mathbf{v}_{\mathbf{b}}^T \frac{\partial \mathcal{L}}{\partial \lambda_b} = \mathbf{V_B}\frac{\partial \mathcal{L}}{\partial \mathbf{\Lambda_B}}\mathbf{V}^T_\mathbf{B}.$$

Using the chain rule, we obtain $$ \frac{\partial \mathcal{L}}{\partial m_i} = \sum_{j,k}\frac{\partial\mathcal{L}}{\partial \mathbf{A}_{jk}} \frac{\partial \mathbf{A}_{jk}}{\partial m_i}=\mathrm{tr}\left[\mathbf{V}^T_\mathbf{B}\frac{\partial \mathcal{L}}{\partial \mathbf{\Lambda_B}}\frac{\partial\mathbf{A}}{\partial m_i}\mathbf{V_B}\right] $$

The problem is, if $\partial \mathcal{L}/\partial \mathbf{V_B} \ne \mathbf{0}$, I cannot find the expression that does not involve the full matrix $\mathbf{A}$ or the full eigenvalues matrix.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.