I am trying to find out the essence of what a determinant is. Besides, in finite dimensions, determinant is the kind of numerical invariant that determines the invertibility of a linear operator, but what about infinite case? Is there a similar invariant? Why if not?

We can define the Fredholm determinant on operators on Hilbert space which differ from the identity by a traceclass operator. This satisfies $$\det(\exp(T)) = \exp(\text{Tr}(T))$$ for traceclass operators $T$. See http://en.wikipedia.org/wiki/Fredholm_determinant 


Attempting to define a wellbehaved "infinitedimensional determinant" for all operators will get us into trouble fairly quickly. Consider, for example, a vector space $V$ of countably infinite dimension and let $A$ be multiplication by some nonzero scalar $\lambda\neq1$. Then $A$ is certainly invertible so its putative determinant should be nonzero. But now upon fixing a basis for $V$ we get the following matrix identities $$ A = \begin{pmatrix} \lambda \\ & \lambda \\ & & \lambda \\ & & & \ddots \end{pmatrix} = \begin{pmatrix} \lambda \\ & 1 \\ & & 1\\ & & & \ddots \end{pmatrix} \begin{pmatrix} 1 \\ & \lambda \\ & & \lambda \\ & & & \ddots \end{pmatrix} = \begin{pmatrix} \lambda \\ & I \end{pmatrix} \begin{pmatrix} 1 \\ & A \end{pmatrix}. $$ If we expect our determinant to behave like its finitedimensional counterpart, then the above would yield $\det A = \lambda \det A$ and consequently that $\det A = 0$, counter to what we expect from the invertibility of $A$. So in attempting to define $\det$ for operators on infinitedimensional spaces, you either have to restrict the class of operators under consideration or lower your expectations of how your $\det$ will be analogous to the finitedimensional one. 


The abstract way to define the determinant of a linear operator $T : V \to V$ of a finitedimensional vector space is that it is the induced action of $T$ on the top exterior power $\Lambda^n(T) : \Lambda^n(V) \to \Lambda^n(V)$, where $n = \dim V$. The top exterior power is $1$dimensional, so $\Lambda^n(T)$ is canonically a scalar. By functoriality of the exterior power, $T \mapsto \Lambda^n(T)$ is a monoid homomorphism, which is why it detects invertibility. So you can see what the problem is in infinite dimensions: if $V$ is infinitedimensional, there is no top exterior power! All the exterior powers are also infinitedimensional. 


There are modifications of the notion of Fredholm determinant for operators on Hilbert space which differ from the identity by an operator from a von NeumannSchatten ideal. A related notion is the one of a von Koch determinant defined for some classes of infinite matrices. For all this see Gohberg, I.C.; Krein, M.G. Introduction to the theory of linear nonselfadjoint operators. Translations of Mathematical Monographs. 18. Providence, RI: American Mathematical Society, 1969. 


The answers above point out that one cannot define a determinant in a meaningful way on the algebra of bounded operators on a Banach space, unless finitedimensional. However, this does not preclude the possibility of doing this for suitable subclasses and this is precisely what Alexander Grothendieck did in his celebrated (amongst functional analysts) article "Théorie de Fredholm" (Bull. Soc. Math. vol. 84freely available online). This is one of those articles which changed the face of mathematics forever. The closely related question of which operators have a trace has been investigated in great detail, by, for example, Albrecht Pietsch and Hermann König. 


To the references already given for the definition and the properties of the determinant $\det(I+T)$, where $T$ is of trace class, I would add: Reed; Simon: Methods of Modern Mathematical Physics IV, 1978. Chapter 13 Taylor, M. E.: Partial Differential Equations I (Basic Theory), 1996. Appendix A. Schmüdgen, K.: Unbounded Selfadjoint Operators on Hilbert Space. Springer GTM, 2011. Ch. 9, sect. 5. 

