Skip to main content
1 of 9
dohmatob
  • 6.9k
  • 1
  • 18
  • 76

Derivative of rank $r$ approximation of matrix

Let $Y \in \mathbb R^{n \times c}$ and $r$ be an integer with $1 \le r \le \min(n,c)$. Consider the problem

$$\text{minimize} \|Y-X\|_{\text{Fro}}^2\text{ over }X \in \mathbb R^{n \times c}\text{ subject to rank}(X) \le r.$$

By the Eckart-Young-Mirsky theorem, one can show that $X = \hat{Y}(r) := Y\Pi(r)$, where $\Pi(r) := V(r)V(r)^T$ the projection onto the subspace spanned by the first $r$ principal vectors $V(r) : = [V_1,\ldots,V_r] \in \mathbb R^{c \times r}$ of $Y$.

Question: What is $\hat{df}(r) := \text{tr}\left(\frac{\partial vec(\hat{Y}(r))}{\partial vec(Y)}\right) = \sum_{i,j,k,l} \frac{\partial \hat{Y}_{ij}(r)}{\partial Y_{kl}}$ ?

Motivation: Such computations are necessary in applying SURE (stein's Unbiased Risk Estimator) theory for tuning penalized least squares models, as a powerful alternative to cross-validation. Here "df" stands for "degrees of freedom" and is a measure of the complexity of the approximator model.

Observation: It's not hard to show that $$\hat{df}(r) \ge \hat{df}_{\text{naive}}(r),$$

where $\hat{df}_{\text{naive}}(r):= nc - (n-r)(c-r)$ is number of free parameters needed to specify an $n$-by-$c$ matrix of rank $r$. Theorem 5.2 of this paper computes the correction $\hat{df}(r) - \hat{df}_{\text{naive}}(r)$ exactly, but the is by direct "brute-force" and involves some very long formulae. I was wondering whether there could be a cleaner way to obtain this.

dohmatob
  • 6.9k
  • 1
  • 18
  • 76