We know that $n\times n$ square Hermitian matrices can be diagonalized and have real eigenvalues.
Suppose I have a countable sized Hermitian matrix $A=(a_{ij})$ where the indices $i$ and $j$ run over the natural numbers and the complex entries satisfy $\bar{a_{ij}}=a_{ji}$.
Can such a matrix $A$ be diagonalized? Otherwise, what conditions can I impose that will make it diagonalizable? I'm interested in the case where only finitely many of the eigenvalues are nonzero.
Such a matrix defines an unbounded operator $\mathcal A$ over the Hilbert space $\ell^2(\mathbb N)$, with domain $D(\mathcal A)$, formed of vectors $x$ such that $\mathcal Ax\in H$. It may happen of course that $\mathcal A$ be a bounded operator. The property $\overline{a_{ij}}=a_{ji}$ tells you that $\langle\mathcal Ax,y\rangle=\langle x,\mathcal Ay\rangle$ whenever $x,y\in D(\mathcal A)$.
For an unbounded operator $\mathcal A$, we define the domain of the adjoint operator $\mathcal A^*$ to be the set of $y\in H$ such that the linear form $x\mapsto\langle\mathcal Ax,y\rangle$ is bounded.
Definition. The operator $\mathcal A$ is self-adjoint if $\langle\mathcal Ax,y\rangle=\langle x,\mathcal Ay\rangle$ whenever $x,y\in D(\mathcal A)$, and in addition $D(\mathcal A^*)=D(\mathcal A)$.
It turns out that the symmetry property does not imply the equality of the domains. The theory of unitary diagonalisation of Hermitian matrices basically extends naturally to self-adjoint operators. See the book by Reed and Simon.
As a comment on Denis's answer... You have to be a little bit careful about what you mean by "diagonalisation" though. Consider $[0,1]$ with Lebesgue measure, and define $T:L^2[0,1]\rightarrow L^2[0,1]$ by $$ T(f)(x) = x f(x). $$ So $T$ is ``multiplication by $x$''. This is easily seen to be self-adjoint. So if we pick an orthonormal basis for $L^2[0,1]$, we get a self-adjoint matrix $A$ of the type you describe (and it induces a bounded map on $\ell^2$ of course).
But notice that $T$ has no eigenvectors or eigenvalues, and thus the same is true of $A$ (assuming we ask for our eigenvectors to be in $\ell^2$, but otherwise I don't see how we can talk about ``eigenvalue'').
Moral: you have to give up "diagonalisation" in favour of the more general idea of being "unitarily equivalent to a (real-valued) multiplication operator".
I'm not sure I understand your question since the answer seems too simple. You must first choose a space of vectors for the matrix to act upon. A natural choice is $\ell^2$, then the natural assumption is that the operator induced by $A$ has a selfadjoint extension, in which case standard spectral theory applies, in particular the spectral theorem gives you what you need. This makes the meaning of 'diagonalizable' precise. Notice that your assumption is that the operator $A$ is symmetric, which is weaker than selfadjoint (but not too much weaker). Other choices are possible of course.
A small preamble. It is rather difficult or impossible to treat the matter in a purely algebraic setting, with no topology. For instance, unless you bound your attention to very particular infinite matrices, you immediately find that the elementary operations on matrices are not even defined, as they would require sums of infinitely many terms. To get a reasonable setting where to extend finite dimensional linear algebra, you need a topology compatible with the linear structure. After all, this is how Functional Analysis was born. In particular, a convenient setting to treat the concept you are talking of, are Hilbert spaces, and the spectral theory of self-adjoint operators, that you can find in any good text on Hilbert spaces (Kato's Perturbation Theory for Linear Operators, to quote one). Here's enough if we succeed in convincing you that Functional Analysis is the right direction to leave the finite dimension; I only recall that the analogue of the notion of eigenvalue becomes more subtle, as there are different reasons why $A-\lambda I$ may fail to be invertible (e.g. still being injective).
