Is this formulation of the Singular Value Decomposition standard? - MathOverflow

Is this formulation of the Singular Value Decomposition standard?

2010-10-24T18:11:51Z

In customary formulations of the Singular Value Decomposition or SVD that I have seen, (e.g., Wikipedia or Gil Strang's textbooks) it is always stated in terms of writing an $m \times n$ matrix $M$ (say of rank $r$) as a product $U \Lambda V$, where $U$ and $V$ are orthogonal $m \times m$ and $n \times n$ matrices and $\Lambda$ is a diagonal $m \times n$ matrix with non-negative "singular values" on the diagonal, $s_1 \ge s_2 \ge \ldots \ge s_r >0$ and the rest zero. Looking back over the many times I have taught linear algebra to both undergraduates and graduate students, I realized that I have not once covered the SVD, and even though I consider myself pretty knowledgeable about linear algebra, I have never felt comfortable with the statement of SVD or felt that I understood it in a more than formal way. (I might add that many theoretical linear algebra texts do not mention the SVD and many of my more theoretically minded colleagues do not even recognize the term.) But a few days ago, a social scientist friend of mine asked me about
a problem he was interested in; one that involved the SVD in an essential way. After thinking about it for a while, I realized that SVD can be reformulated as a statement about linear transformations that, to me at least, seems a lot more conceptual and geometric:

If $T$ is a linear map, say of rank $r$, between finite dimensional inner-product spaces $V$ and $W$, then there are orthonormal bases $v_1, \ldots, v_m$ for $V$ and $w_1, \ldots, w_n$ for $W$ and $r$ positive numbers $s_1 \ge s_2 \ge \ldots \ge s_r$, such that $T v_i$ equals $s_i w_i$ if $i \le r$ and equals zero if $i > r$.

I certainly realize that this is a pretty obvious reformulation of SVD, once you see it (and
those poor misguided souls who prefer matrices to linear transformations may even see it as a step backwards :-), but my question is whether there is some standard source for this reformulation that I can reference.

Answer by Mark Meckes for Is this formulation of the Singular Value Decomposition standard?

2010-10-24T18:24:27Z

This formulation, or something very close to it (I don't have the book with me) is in Axler's Linear Algebra Done Right.

Answer by Angelo for Is this formulation of the Singular Value Decomposition standard?

2010-10-24T19:14:18Z

I just looked in Wikipedia (http://en.wikipedia.org/wiki/Singular_value_decomposition). There is a very thorough discussion, including a section "Geometric meaning" in which your interpretation is clearly explained. Well, there $K^n$, where $K = \mathbb R$ or $K = \mathbb C$ is used, instead of arbitrary inner product spaces, but I suppose you'll agree that the difference is not essential.

Answer by Ramsay for Is this formulation of the Singular Value Decomposition standard?

2010-10-24T20:37:23Z

The book "Numerical Linear Algebra" by Trefethen and Bau, also introduces the SVD in this way. The relevant chapters (4 and 5) seem to be complete in Google books. The exposition is not tainted by the 'numerical' in the title.

I don't understand why the SVN isn't given more emphasis in standard introductions to linear algebra. It seems to give a good intuitive decomposition of a linear operator. I would expect it to be a useful theoretical tool even if numerical issues are not being considered.

Answer by Denis Serre for Is this formulation of the Singular Value Decomposition standard?

2010-11-10T09:59:24Z

The singular values do have a sound geometrical meaning. The first one is nothing but the operator norm of $T$, that is the maximum dilation coefficient $$\frac{\|Tx\|}{\|x\|}.$$ The next ones can be seen also as maximum of dilation coefficients, provided you replace lines by subspaces. For instance $$s_k=\max_{\dim F=k}\min\left\{\frac{\|Tx\|}{\|x\|};x\in F,x\ne0\right\}.$$ In terms of the exterior algebra over $\mathbb C^k$ ($k=m$ or $n$), you have $$s_1\cdots s_k=\sup\left\{\frac{\|Tx_1\wedge\cdots\wedge Tx_k\|}{\|x_1\wedge\cdots\wedge x_k\|};x_1,\ldots,x_k\in F\right\}.$$

Notice also that there is a $p$-adic version of the SVD decomposition. See K. S. Kedlaya. $p$-adic differential equations. Cambridge Studies in Advanced Mathematics, 125. Cambridge University Press, Cambridge, 2010

Edit. I should have also given the formula $$s_1+\cdots+s_k=\max\{{\rm Tr}(PTQ); P\hbox{ and } Q \hbox{ are unitary projectors of rank } k\}.$$ This has the interesting consequence that $T\mapsto s_1+\cdots+s_k$ is a convex function.

Edit (bis). Actually, $T\mapsto s_1\cdots s_k$ is rank-one convex. This means that it is convex along every line $A+{\mathbb R} B$ for which $B-A$ is a rank-one matrix.

Answer by wildildildlife for Is this formulation of the Singular Value Decomposition standard?

2010-11-10T17:21:21Z

It's in my favourite linear algebra book Advanced Linear Algebra by Steven Roman, chapter 17 (called Singular Values and the Moore-Penrose inverse).