I am trying to find out the essence of what a determinant is. Besides, in finite dimensions, determinant is the kind of numerical invariant that determines the invertibility of a linear operator, but what about infinite case? Is there a similar invariant? Why if not?

1$\begingroup$ If $V$ is a vector space and $f$ is an endomorphism of $V$, then sometimes a determinant $\det f$ can be defined even without requiring $V$ to be finitedimensional. For example, if there is a finitedimensional vector subspace $W$ of $V$ such that $f\left(V\right)\subseteq W$, then we can define $\det f$ to be the determinant of the map $W\to W$ obtained by restricting $f$; this will actually be independent on $W$ (as long as $W$ satisfies $f\left(V\right)\subseteq W$). But this is a rather trivial case (although surprisingly useful in Ktheory). $\endgroup$– darij grinbergApr 4, 2013 at 1:33

$\begingroup$ If $V$ is Banach(?), there might be a more general type of endomorphisms $f$ for which $\det f$ could be defined in some reasonable way; "determinantclass" endomorphisms, so to speak (in analogy to traceclass endomorphisms). In the general case, however, determinants make no sense in infinite dimensions. This shouldn't be very surprising since, e. g., an injective endomorphism of an infinitedimensional space can fail to be surjective, and vice versa. $\endgroup$– darij grinbergApr 4, 2013 at 1:36

2$\begingroup$ For me, "the essence of what a determinant is" is the measure of the distortion of volume. But volume is a complicated notion in infinite dimensions. $\endgroup$– Claudio GorodskiApr 4, 2013 at 1:51

2$\begingroup$ @darij: The condition you described only lets you define $\det (1Tf)$. This will be a polynomial whose roots are the reciprocals of the nonzero eigenvalues of $f$. $\endgroup$– Kevin VentulloApr 4, 2013 at 2:42

$\begingroup$ Oh, right! Yeah, that's what I meant. $\endgroup$– darij grinbergApr 4, 2013 at 3:53
6 Answers
We can define the Fredholm determinant on operators on Hilbert space which differ from the identity by a traceclass operator. This satisfies $$\det(\exp(T)) = \exp(\text{Tr}(T))$$ for traceclass operators $T$. See http://en.wikipedia.org/wiki/Fredholm_determinant

2$\begingroup$ Thank you Professor Israel. Now everything is perfectly understandable! $\endgroup$– XuxuApr 6, 2013 at 2:21
Attempting to define a wellbehaved "infinitedimensional determinant" for all operators will get us into trouble fairly quickly. Consider, for example, a vector space $V$ of countably infinite dimension and let $A$ be multiplication by some nonzero scalar $\lambda\neq1$. Then $A$ is certainly invertible so its putative determinant should be nonzero. But now upon fixing a basis for $V$ we get the following matrix identities $$ A = \begin{pmatrix} \lambda \\ & \lambda \\ & & \lambda \\ & & & \ddots \end{pmatrix} = \begin{pmatrix} \lambda \\ & 1 \\ & & 1\\ & & & \ddots \end{pmatrix} \begin{pmatrix} 1 \\ & \lambda \\ & & \lambda \\ & & & \ddots \end{pmatrix} = \begin{pmatrix} \lambda \\ & I \end{pmatrix} \begin{pmatrix} 1 \\ & A \end{pmatrix}. $$ If we expect our determinant to behave like its finitedimensional counterpart, then the above would yield $\det A = \lambda \det A$ and consequently that $\det A = 0$, counter to what we expect from the invertibility of $A$.
So in attempting to define $\det$ for operators on infinitedimensional spaces, you either have to restrict the class of operators under consideration or lower your expectations of how your $\det$ will be analogous to the finitedimensional one.
I am trying to find out the essence of what a determinant is.
The abstract way to define the determinant of a linear operator $T : V \to V$ of a finitedimensional vector space is that it is the induced action of $T$ on the top exterior power $\Lambda^n(T) : \Lambda^n(V) \to \Lambda^n(V)$, where $n = \dim V$. The top exterior power is $1$dimensional, so $\Lambda^n(T)$ is canonically a scalar. By functoriality of the exterior power, $T \mapsto \Lambda^n(T)$ is a monoid homomorphism, which is why it detects invertibility.
So you can see what the problem is in infinite dimensions: if $V$ is infinitedimensional, there is no top exterior power! All the exterior powers are also infinitedimensional.

$\begingroup$ [deleted some earlier comments as they are addressed in the Wikipedia page Robert Israel links to] $\endgroup$ Apr 4, 2013 at 6:17

2$\begingroup$ For $T$ sufficiently close to the identity, one might hope for a reasonable way to define an action of $T$ on an "infinite exterior power" $\Lambda^{\dim V}(V)$ (there is an infinite product that has a hope of converging), and I guess this is what the Fredholm determinant makes precise. $\endgroup$ Apr 4, 2013 at 6:17

1$\begingroup$ Thank you so much Qiaochu. It just occurred to me that I've run into the stuff of viewing determinant as actions on top exterior forms when studying smooth manifolds, but your comments for infinite case is really inspiring and I really appreciate your help! $\endgroup$– XuxuApr 6, 2013 at 2:19
There are modifications of the notion of Fredholm determinant for operators on Hilbert space which differ from the identity by an operator from a von NeumannSchatten ideal. A related notion is the one of a von Koch determinant defined for some classes of infinite matrices. For all this see
Gohberg, I.C.; Krein, M.G. Introduction to the theory of linear nonselfadjoint operators. Translations of Mathematical Monographs. 18. Providence, RI: American Mathematical Society, 1969.
The answers above point out that one cannot define a determinant in a meaningful way on the algebra of bounded operators on a Banach space, unless finitedimensional. However, this does not preclude the possibility of doing this for suitable subclasses and this is precisely what Alexander Grothendieck did in his celebrated (amongst functional analysts) article "Théorie de Fredholm" (Bull. Soc. Math. vol. 84freely available online). This is one of those articles which changed the face of mathematics forever. The closely related question of which operators have a trace has been investigated in great detail, by, for example, Albrecht Pietsch and Hermann König.

$\begingroup$ I believe this is related to the reference Robert Israel gives for the Fredholm determinant? (And I guess this was what led Grothendick to the eponymous AP?) $\endgroup$ Apr 4, 2013 at 6:41
To the references already given for the definition and the properties of the determinant $\det(I+T)$, where $T$ is of trace class, I would add:
Reed; Simon: Methods of Modern Mathematical Physics IV, 1978. Chapter 13
Taylor, M. E.: Partial Differential Equations I (Basic Theory), 1996. Appendix A.
Schmüdgen, K.: Unbounded Selfadjoint Operators on Hilbert Space. Springer GTM, 2011. Ch. 9, sect. 5.