Suppose we have an infinite matrix A = (aij) (i, j positive integers). What is the "right" definition of determinant of such a matrix? (Or does such a notion even exist?) Of course, I don't necessarily expect every such matrix to have a determinant -- presumably there are questions of convergence -- but what should the quantity be? The problem I have is that there are several ways of looking at the determinant of a finite square matrix, and it's not clear to me what the "essence" of the determinant is.
There is a class of linear operators that have a determinant. They are, for some strange reason, known as "operators with a determinant".
For Banach spaces, the essential details go along these lines. Fix a Banach space, X, and consider the finite rank linear operators. That means that T: X → X is such that Im(T) is finite dimensional. Such operators have a well-defined trace, tr(T). Using this trace we can define a norm on the subspace of finite-rank operators. If our operator were diagonalisable, we would define it as the sum of the absolute values of the eigenvalues (of which only finitely many are non-zero, of course). This norm is finer than the operator norm. We then take the closure in the space of all operators of the space of finite-rank operators with respect to this trace norm. These operators are called trace class operators. For such, there is a well-defined notion of a trace.
(Incidentally, these operators form a two-sided ideal in the space of all operators and are actually the dual of the space of all operators via the pairing (S,T) → tr(ST).)
Now trace and determinant are very closely linked via the forumula etr T = det eT. This means that we can use our trace class operators to define a new class of "operators with a determinant". The key property should be that the exponential of a trace class operator should have a determinant. This suggests looking at the family of operators which differ from the identity by a trace class operator. Within this, we can look at the group of units, that is invertible operators.
So an "operator with a determinant" is an invertible operator that differs from the identity by one of trace class.
For more details, I recommend the book "Trace ideals and their applications" by Barry Simon (MR541149) and the article "On the homotopy type of certain groups of operators" by Richard Palais (MR0175130).
But defining the determinant of an arbitrary operator is, of course, impossible. One can always figure out a renormalisation for a particular operator but there just ain't gonna be a system that works for everything: obviously det(I) = 1 but then det(2I) = ?
(I should also say that I picked Banach spaces for ease of exposition. One can generalise this to locally convex topological spaces, but that involves handling nuclear materials so caution is advised.)
There are a lot of subtleties that you need to watch out for. First of all, "infinite matrices" aren't well-defined as linear transformations without additional hypotheses. A typical case in combinatorics is that the matrix is triangular and you're only interested in how it acts on a space of formal power series; the t-adic topology is what gives you convergence here. A typical case in analysis is that you're describing a bounded linear operator between separable Hilbert spaces, and then there is the notion of orthonormal basis to work with. In any case you need a topology on the underlying vector space to make sense of infinite sums.
If you define the determinant of a matrix to be the product of its eigenvalues, then you run into immediate trouble: "infinite matrices" don't necessarily have any, even over an algebraically closed field. And in the nicest case, e.g. compact self-adjoint, the eigenvalues tend to zero and their product is zero. I also believe one can show that there is no nontrivial continuous homomorphism GL(H) -> C for H a Hilbert space. Finally, if you think of the determinant in terms of exterior powers, then it's not hard to see that for an infinite-dimensional space H, however you want to define the exterior powers of H they should always be infinite-dimensional.
Having said all that, there is a notion of regularized determinant in the literature, but I'm afraid I couldn't tell you anything about it.