## Why do we want to have orthogonal bases in decompositions?

In the decompositions I encountered so far, we all had orthogonal set of bases. For example in Singular Value Decomposition, we had orthogonal singular right and left vectors, in [discrete] cosine transform (or [discrete] fourier transform) we had again orthogonal bases.

To describe any vector $x \in \mathbb{C}^N$, we need to have $N$ independent set of basis vectors but independent doesn't necessarily mean orthogonal. My intentions behind selecting orthogonal vectors are as follows:

• The solution is not unique for $x$ if the basis are not orthogonal.
• It is easy to find the solution numerically by projecting $x$ onto each vector and this solution doesn't depend on the order of the bases. Otherwise, it depends on the order.
• If we are talking about some set of vectors, they might be correlated in the original space, but uncorrelated in the transformd space which might be important when analyzing the data, in dimensionality reduction or compression.

I'm trying to understand the big picture. Do you think that I am right with these? Do you have any suggestions, what is the main reason for selecting orthogonal bases?

-
Frequently you don't (or can't) use orthonormal basis in decompositions. It begs the question of what a decomposition is for I suppose. If you're interested in the dynamical properties of the matrices -- like understanding $A^k$ for all $k$, and if say $A$ is an integer matrix you'll probably want your conjugating matrices to be integer matrices as well, so it's very unlikely they'll be orthogonal matrices. – Ryan Budney Oct 13 2010 at 18:03
One answer to this is that in many situations, orthogonal elements represent elements of a system that are in some sense non-interacting, so that in solving a problem they can be handled independently. This is the case in particular when the elements are the eigendirections of a self-adjoint operator. – Dick Palais Oct 13 2010 at 18:08

Your first point, non-uniqueness, is definitely false. One of the basic facts in linear algebra is precisely that for any fixed set of basis vectors (we don't even have to work on a vector space endowed with an inner product, so orthogonality doesn't come in at all), a given vector has a unique decomposition.

For if that were not true, let the basis vectors be $e_1, \ldots, e_n$, then you have two sets of numbers $a_1, \ldots a_n$ and $b_1, \ldots, b_n$ such that

$$a_1 e_1 + \cdots + a_n e_n = x = b_1 e_1 + \cdots + b_n e_n \implies (a_1 - b_1) e_1 + \cdots + (a_n - b_n) e_n = 0$$

if the sets $a_*$ and $b_*$ are not identical, this implies that $e_1\ldots e_n$ are linearly dependent, contradicting the assumption that they form a basis.

The second point, however, is a biggie. Without an inner product you cannot define an orthgonal projection. Now, generally, this is not too much of a problem. Given the basis vectors $e_1,\ldots, e_n$, finding the coordinates $a_1,\ldots, a_n$ of a given vector $x$ in this basis is just solving a linear system of equations, which actually is not too hard numerically, in finite dimensional systems. In infinite dimensional systems this inversion of the transformation matrix business gets slightly tricky.

The key is to note that without using the orthogonal projection, you cannot answer the question "what is the length of $x$ in the direction of $e_1$?" without knowing the entire set of basis vectors. (Remember that without using the orthogonal projection, you need to solve a linear system of equations to extract the coordinates; if you only specify one of the basis vectors, you do not have a complete system of equations, and the solution is underdetermined. I suspect this is what you had in the back of your mind for the first point.) This is actually a very fundamental fact in geometry, regarding coordinate systems. (I've once heard this described as the "second fundamental mistake in multivariable calculus" made by many students.)

Using the orthogonal projection/the inner product, you can answer the question, as long as you allow only orthgonal completions of the basis starting from the known vector. This fact is immensely useful when dealing with infinite dimensional systems.

I also don't quite like your formulation of the third point. A vector is a vector is a vector. It is independent of the coordinate representation. So I'd expect that if two vectors are correlated (assuming correlation is an intrinsic property), they better stay correlated without regard to choice of bases. What is more reasonable to say is that two vectors maybe uncorrelated in reality, but not obviously so when presented in one particular coordinate system, whereas the lack of correlation is immediately obvious when the vectors are written in another basis. But this observation has rather little to do with orthogonality. It only has some relation to orthogonality if you define "correlation" by some inner product (say, in some presentations of Quantum Mechanics). But then you are just saying that orthogonality of two vectors are not necessarily obvious, except when they are.

My personal philosophy is more one of practicality: the various properties of orthogonal bases won't make solving the problem harder. So unless you are in a situation where those properties don't make solving the problem easier, and some other basis does (like what Ryan Budney described), there's no harm in prefering an orthogonal basis. Furthermore, as Dick Palais observed above, one case where an orthogonal bases really falls out naturally at you is the case of the spectral theorem. The spectral theorem is, in some sense, the correct version of your point 3, that in certain situations, there is a set of basis vectors that are mathematically special. And this set happens to always be orthogonal.

Edit A little more about correlation. This is what I like to tell students when studying linear algebra. A vector is a vector. It is an object, not a bunch of numbers. When I hand you a rectangular box and ask you how tall the box is, the answer depends on which side is "up". This is similar to how you should think of the coordinate values of a vector inside a basis: it is obtained by a bunch of measurements. (Picking which side is "up" and measuring the height in that direction, however, in a non-orthogonal system, will require knowing all the basis vectors. See my earlier point.)

The point is that to quantitatively study science, and to perform numerical analysis, you can only work with numbers, not physical objects. So you have to work with measurements. And in your case, the correlation you are speaking of is correlation between the measurements of (I suppse) different "properties" of some object. And since what and how you measure depends on which basis you choose, the correlation between the data will also depend on which basis you choose. If you pick properties of an object that are correlated, then your data obtained from the measurements will also be correlated. The PCA you speak of is a way to disentangle that. It may be difficult to determine whether two properties of an object is correlated. Maybe the presence of a correlation is what you want to detect. The PCA tells you that, if there were in fact two independent properties of an object, but you just chose a bad set of measurements so that the properties you measure do not "line up" with the independent properties (that the properties you measure have a little bit of each), you can figure it out with a suitable transformation of the data at the end of the day. So you don't need to worry about choosing the best set of properties to measure.

-
 Thank you for the simple proof for my first point. You are definitely right. I didn't think on it clearly and accepted it as correct when a a friend of mine told that it should be that way. For the third point, I will check what spectral theorem gives us. Actually, the correlation/dependence was in my mind because of principal component analysis which is close to singular value decomposition. For example, it this image (bit.ly/cyt22i), when we change the bases, the data becomes independent (but for this gaussian distributed samples in fact). Is correlation independent from the bases? – İsmail Arı Oct 14 2010 at 9:41 I see, I didn't quite understand what you mean by correlation. Let me add one more paragraph in the bottom. – Willie Wong Oct 14 2010 at 13:46 I think I understood the mathematical way of seeing vectors. I always thought of them as being a bunch of numbers actually because of my engineering-oriented education. Thank you for the additional explanation. – İsmail Arı Oct 14 2010 at 19:45

I assume you are seeing things from a numerical point of view, since you only mentioned computation-oriented decompositions.

So here's a computational motivation: orthogonal matrices have condition number 1, thus multiplying and dividing by them is a numerically tame operation that does not increase the (norm-wise) errors. For instance, even when you work with symplectic matrices, you usually look for symplectic and orthogonal matrices.

-

Maybe some examples of non-orthogonal decompositions are helpful: In the context of wavelet one frequently uses bi-orthonormal systems, i.e. two sets of vectors $(u_i)$ and $(v_i)$ such that $\langle u_i,v_j\rangle = \delta_{i,j}$. In this setting the calculations of coefficients can be done stably and efficiently. In real life, such bi-orthogonal wavelet bases are at the heart of the JPEG 2000 compression standard.

Another important notion in this context is the notion of frames.

-

Well, when multiplying a vector ${\bf x}$ (say of norm 1) with a matrix ${\bf A}$ you are expecting to get a vector that lies on the surface of the ellipse $\left\|{\bf A}{\bf x}\right\|^2_2$. The directions of the axes of this ellipse are the eigenvectors and the eigenvalues dictate the equatorial radii. So every time you multiply a vector with that matrix, this decomposition into axes and radii tell you how your matrix distorts an "input" vector. That might be one pictorial reason :)

-