Suppose we have an infinite matrix A = (a_{ij}) (i, j positive integers). What is the "right" definition of determinant of such a matrix? (Or does such a notion even exist?) Of course, I don't necessarily expect every such matrix to have a determinant  presumably there are questions of convergence  but what should the quantity be? The problem I have is that there are several ways of looking at the determinant of a finite square matrix, and it's not clear to me what the "essence" of the determinant is.

$\begingroup$ I take it you're not just talking about matrices that are eventually 1 or 0 on the diagonal. $\endgroup$ – Josh Oct 22 '09 at 19:45

$\begingroup$ Not necessarily. I don't have any particular restrictions (or lack thereof) in mind, though I would be interested to find out what you can say when you impose certain restrictions. $\endgroup$ – Gabe Cunningham Oct 22 '09 at 22:26
There is a class of linear operators that have a determinant. They are, for some strange reason, known as "operators with a determinant".
For Banach spaces, the essential details go along these lines. Fix a Banach space, X, and consider the finite rank linear operators. That means that T: X → X is such that Im(T) is finite dimensional. Such operators have a welldefined trace, tr(T). Using this trace we can define a norm on the subspace of finiterank operators. If our operator were diagonalisable, we would define it as the sum of the absolute values of the eigenvalues (of which only finitely many are nonzero, of course). This norm is finer than the operator norm. We then take the closure in the space of all operators of the space of finiterank operators with respect to this trace norm. These operators are called trace class operators. For such, there is a welldefined notion of a trace.
(Incidentally, these operators form a twosided ideal in the space of all operators and are actually the dual of the space of all operators via the pairing (S,T) → tr(ST).)
Now trace and determinant are very closely linked via the forumula e^{tr T} = det e^{T}. This means that we can use our trace class operators to define a new class of "operators with a determinant". The key property should be that the exponential of a trace class operator should have a determinant. This suggests looking at the family of operators which differ from the identity by a trace class operator. Within this, we can look at the group of units, that is invertible operators.
So an "operator with a determinant" is an invertible operator that differs from the identity by one of trace class.
For more details, I recommend the book "Trace ideals and their applications" by Barry Simon (MR541149) and the article "On the homotopy type of certain groups of operators" by Richard Palais (MR0175130).
But defining the determinant of an arbitrary operator is, of course, impossible. One can always figure out a renormalisation for a particular operator but there just ain't gonna be a system that works for everything: obviously det(I) = 1 but then det(2I) = ?
(I should also say that I picked Banach spaces for ease of exposition. One can generalise this to locally convex topological spaces, but that involves handling nuclear materials so caution is advised.)

3$\begingroup$ It is not a good idea to try to define trace class operators on arbitrary Banach spaces, it is better to restrict yourself to Hilbert spaces. Indeed, there is an issue in your claim "We then take the closure in the space of all operators of the space of finiterank operators with respect to this trace norm." What you can do is take the completion, say $P$ for "projective", of the finite rank operators for the (correctly defined) trace norm. On $P$ there is a welldefined notion of trace. There is also a norm $1$ map $P \to B(X)$, and its images are called the nuclear operators... $\endgroup$ – Mikael de la Salle Feb 6 '13 at 15:19

3$\begingroup$ ... (continued) but the problem is that this map is not injective in general. This map is injective if and only if $X$ has the Approximation Property (AP) ( en.wikipedia.org/wiki/Approximation_property ). Hence it is not possible to define a reasonable notion of trace on the nuclear operators on a Banach space without AP. This difficulty is completely overlooked in Simon's book. By the way, the wikipedia page on AP also contains mistakes, I will have to clean it when I have time if nobody does it before. $\endgroup$ – Mikael de la Salle Feb 6 '13 at 15:24

4$\begingroup$ It should perhaps also be pointed out that in the Hilbert space case, this notion of determinant is known as the Fredholm determinant: en.wikipedia.org/wiki/Fredholm_determinant $\endgroup$ – Terry Tao Feb 6 '13 at 18:32
There are a lot of subtleties that you need to watch out for. First of all, "infinite matrices" aren't welldefined as linear transformations without additional hypotheses. A typical case in combinatorics is that the matrix is triangular and you're only interested in how it acts on a space of formal power series; the tadic topology is what gives you convergence here. A typical case in analysis is that you're describing a bounded linear operator between separable Hilbert spaces, and then there is the notion of orthonormal basis to work with. In any case you need a topology on the underlying vector space to make sense of infinite sums.
If you define the determinant of a matrix to be the product of its eigenvalues, then you run into immediate trouble: "infinite matrices" don't necessarily have any, even over an algebraically closed field. And in the nicest case, e.g. compact selfadjoint, the eigenvalues tend to zero and their product is zero. I also believe one can show that there is no nontrivial continuous homomorphism GL(H) > C for H a Hilbert space. Finally, if you think of the determinant in terms of exterior powers, then it's not hard to see that for an infinitedimensional space H, however you want to define the exterior powers of H they should always be infinitedimensional.
Having said all that, there is a notion of regularized determinant in the literature, but I'm afraid I couldn't tell you anything about it.

$\begingroup$ There are many regularization procedures. One popular one in the physics literature is called "zetafunction regularization". I don't particularly understand it, but I posted the definition and some questions about it at mathoverflow.net/questions/183/… $\endgroup$ – Theo JohnsonFreyd Oct 22 '09 at 16:29