It seems to me that a lot of people do deformation theory (of schemes, sheaves, maps etc) in derived category (of an appropriate abelian category). For example, the cotangent complex of a morphism $f:X\rightarrow Y$ of schemes is defined as an object in the derived category of coherent sheaves $X$. I would like to understand why derived category is an appropriate category to do deformation theory. I would appreciate it if someone could give me a good example or motivation.

8$\begingroup$ I feel puzzled about all "why do we need" questions. I think all these questions can be answered in the same way: if you don't know why, then you don't need it. $\endgroup$– Fernando MuroCommented Nov 1, 2012 at 1:37

49$\begingroup$ Fernando: That seems to be rather a hard line to take. Clearly, many people ask questions like that when they feel like something may be useful to them, but they're not sure enough to devote a lot of time to studying it. In such a situation, isn't it quite helpful to hear friendly explanations of why other people find it useful? $\endgroup$– Minhyong KimCommented Nov 1, 2012 at 14:28

28$\begingroup$ And then other times, some popular thing looks vaguely interesting, even if you're not motivated by too specific a need. Then you'd like to hear what all the fuss is about in a relatively painless way. $\endgroup$– Minhyong KimCommented Nov 1, 2012 at 14:32

15$\begingroup$ @Fernando Muro: That's fine of course. I know a number of great mathematicians who essentially share your view as a matter of personal practice. I was just puzzled that you were puzzled by some other people having the view I outlined! Clearly these different temperaments can coexist and contribute to mathematics. $\endgroup$– Minhyong KimCommented Nov 2, 2012 at 0:22

20$\begingroup$ @Fernando Muro I hope you don't mind if I revise my statement into a slightly more presumptuous form. I know a number of great mathematicians who share your view as a matter of personal conviction. My impression is that they find it rather hard to follow consistently as a matter of actual practice. $\endgroup$– Minhyong KimCommented Nov 2, 2012 at 0:28
3 Answers
I'll post an answer to what I think is a reasonable question, hoping that someone more expert will improve on this answer. My apologies if this answer is too chatty.
First, a very simple reason you might hope that there is something like a cotangent complex and that it should be an object in the derived category of quasicoherent sheaves. Given a morphism $f: X \rightarrow Y$ of schemes (over some base, which I leave implicit), you get a right exact sequence of quasicoherent sheaves on $X$:
$$f^{*}\Omega^{1}_{Y} \rightarrow \Omega^{1}_{X} \rightarrow \Omega^{1}_{X/Y} \rightarrow 0.$$
Experience has taught us that when we have a functorial halfexact sequence, it can often be completed functorially to a long exact sequence involving 'derived functors', and in abelian contexts such a long exact sequence is usually associated to a short exact sequence (or exact triangle) of 'total derived functor' complexes, the complexes being welldefined up to quasiisomorphism and hence objects in a derived category. This is the semimodern point of view on derived functors, which one can learn for instance from GelfandManin's Methods of Homological Algebra. Experience has also shown that not only is the cohomology of the total derived functor complex important in computations, but that the complex itself, up to quasiisomorphism, contains strictly more information and is often easier to work with, until the very last moment when you want to compute some cohomology.
Once you've seen this work a number of times (say for global sections, for $Hom$, and for $\otimes$), one might ask if there is a total derived functor of $\Omega^{1}$, call it $\mathbb{L}$, which among other things produces an exact triangle
$$f^{*}\mathbb{L}_{Y} \rightarrow \mathbb{L}_{X} \rightarrow \mathbb{L}_{X/Y}$$
such that the long exact sequence of cohomology sheaves begins with the original right exact sequence.
If you believe that such a thing should be useful, then you might go about trying to construct it. One way to do this involves interpreting $\Omega^{1}$ as representing derivations which in turn correspond to squarezero extensions, and this is where deformation theory comes in. So one might begin to ask what 'derived squarezero extensions' should be, and you might guess that you should try to extend not just by modules but by bounded above complexes of modules. When you do this, such a squarezero extension becomes not just a commutative algebra but some kind of derived version thereof, such as a simplicial commutative algebra. In these terms, the cotangent complex $\mathbb{L}$ of a commutative algebra turns out to be nothing but K\"ahler differentials of an appropriate resolution of our commutative algebra in the world of simplicial commutative algebras. Following this idea through and figuring out how descent should work leads, after a long song and dance, to the desired theory of the cotangent complex.
Once you set this all up, it becomes clear that there was no reason to restrict oneself to classical commutative algebras in setting up algebraic geometry, but that one could have worked with simplicial commutative algebras to begin with, and this leads to `derived algebraic geometry'. In some sense, this is the natural place in which to understand the cotangent complex, and here the higher cohomology of the cotangent complex has a natural geometric interpretation.
One should also point out that Quillen's point of view on the cotangent complex was as a homology theory for commutative algebras (search for AndreQuillen homology), which in a precise sense is an analogue of the usual homology of a topological space. This is described in the last chapter of Quillen's Homotopical Algebra and is also discussed in GoerssSchemmerhorn's Model Categories and Simplicial Methods.
To summarise. Deformation theory is about squarezero extensions (as well as about other more general infinitesimal extensions). K\"ahler differentials corepresent derivations which in turn correspond to squarezero extensions. For each morphism of schemes $X \rightarrow Y$, there is a natural right exact sequence involving K\"ahler differentials, which it would be useful to complete to a long exact sequence. Even better, we'd like this long exact sequence to come from an exact triangle of objects in the derived category of quasicoherent sheaves. Realising this goal naturally leads to derived or homotopical algebraic geometry.

$\begingroup$ A typo in the triangle of cotangent complex... $\endgroup$– tempCommented Nov 3, 2012 at 14:47

$\begingroup$ And the cotangent complex is not constructed as an object in the derived category of quasicoherent sheaves, it is an object in the derived category of O_X modules whose cohomologies are quasicoherent. Sorry for being picky. $\endgroup$– tempCommented Nov 3, 2012 at 14:50

$\begingroup$ @temp: are you really being picky, or just thinking of a particular realisation of the cotangent complex? it is my impression that it can be defined entirely internally to QC, at least if you go full derived. $\endgroup$ Commented Nov 4, 2012 at 0:36

3$\begingroup$ @temp Typo corrected. Thank you. To expand on your pickiness, for people who haven't encountered this before, there are a priori two different categories that you might consider: localisation at quasiisomorphisms of the category of complex of quasicoherent sheaves or the subcategory with quasicoherent cohomology of the localisation at quasiisomorphisms of complexes of
$O_X$
modules. If a scheme is not quasicompact, these can be different, and it is usually the latter that one is interested in. Jacob Bell's comment is that there are other ways to realise the derived category... $\endgroup$ Commented Nov 5, 2012 at 8:50 
$\begingroup$ ..., for example by basically defining it via descent on affines. Full derived here perhaps means that in order to have descent for complexes up to quasiisomorphism, you have to do descent with homotopical wisdom. $\endgroup$ Commented Nov 5, 2012 at 8:52
Perhaps one way of reading this question is "Why is it important to think about complexes when doing deformation theory?" One must certainly accept that deformation theory is cohomological in nature: consider the standard Čech construction of classes in $H^1(X, T_X)$ and $H^2(X, T_X)$ corresponding respectively to deformations and obstructions to deformations of a smooth scheme $X$; or consider the deformations of a regular embedding $X \rightarrow Y$, which are parameterized by $H^0(X, N_{X/Y})$ and obstructed by $H^1(X, N_{X/Y})$.
Let me recall the arguments that yield these classes. Locally a smooth scheme has only one deformation up to isomorphism, so global deformations arise from gluing the local ones; the gluing data come from the tangent bundle, so deformations are classified by $H^1(X, T_X)$. Similarly, solutions to an obstructed deformation problem exist locally and are locally unique up to isomorphism, so the obstruction comes from gluing. One writes down the gluing condition and observes that it is a Čech $2$cocycle (or observes directly that deformations form a gerbe...).
The argument for a regular embedding is similar. One first gives a direct construction for the identification of $H^0(X,N_{X/Y})$ and the deformations of the embedding; then one shows that there are no local obstructions, so global obstructions arise entirely from gluing, and this gives a Čech 1cocycle with values in $N_{X/Y}$.
What about a more general deformation problem? For example, suppose $X$ is a local complete intersection. A great deal of the arguments above goes through without modification: locally, deformations of a local complete intersection are unobstructed, so obstructions should come from gluing. One should be able to produce a Čech cocycle obstructing the existence of a global deformation. But a cocycle valued in what?
The answer is the (shifted) tangent complex (dual to the cotangent complex mentioned in another answer). This is a complex in in degrees $[0,1]$ that coincides with $T_X$ if $X$. The relative tangent complex of a regular embedding $X \rightarrow Y$ is $N_{X/Y}[1]$. Thus the tangent complex recovers the two special cases discussed above.
For a complete intersection $X$ in a smooth scheme $Y$, the tangent complex of $X$ may be constructed as $[T_Y \vert_X \rightarrow N_{X/Y}]$. If $X$ is only a local complete intersection then this construction works locally in $X$, but it's not going to glue to something global. However, the local construction is quasiisomorphic to the restriction of something global. At last, the derived category appears.
All of this is vastly generalized by the cotangent complex, which works the same way as above for arbitrary schemes, without the lci restriction. You can read more about that in Chris Brav's answer.
I do not know about the general deformation theory (if there is such a thing), so I will talk about the special case I am familiar with, namely, representation varieties $R=Hom(\pi,G)$ of representations of finitelygenerated groups $\pi$ to a Lie group $G$. In this case, as in some other cases, each analytical germ $(R,r)$ of $R$, regarded as the "deformation space" of $r$, is completely determined by an appropriate controlling differential graded Lie algebra (dgla) $A^\bullet$. A dgla $A^\bullet$ is a certain chain complex equipped with a binary operation called Lie bracket, satisfying certain axioms. The reason for appearance of dgla's in the context of representation varieties $R$ is the basic fact that representations of finitely generated groups could be identified with holonomies of flat connections on appropriate bundles, which, in turn, are described by differential forms with appropriate coefficients. For the differential forms one has both the exterior differential and a bracket, coming from the bracket of the Lie algebra of $G$, hence, one gets a dgla.
Then, there is a basic theorem observed by Deligne, but going back to the earlier work of Schlessinger and Stasheff:
Equivalence Theorem. A weak isomorphism of dglas induces an isomorphism of analytical germs of deformation spaces.
A proof of this theorem (in the general context of dglas, not only for the ones corresponding to representation varieties) could be found in this paper "The Deformation Theory of Representations of Fundamental Groups of Compact Kaehler Manifolds", Publ. Math. I.H.E.S., 67(1988) by Goldman and Millson. The original letter from Deligne could be found here.
Definition. Two dglas $A^\bullet, B^\bullet$ are weakly isomorphic if there is a morphism $f: A^\bullet \to B^\bullet$ which induces an isomorphism on the level of $H^0, H^1$ and an epimorphism on the level of $H^2$.
This, of course, holds if $f$ induces an isomorphism of all cohomology groups, i.e., is a quasiisomorphism of dglas. This means that the derived category of dglas is the natural framework for the deformation theory whenever it is determined by dglas, since one can expoit sequences $$ A^\bullet \rightarrow B^\bullet \leftarrow C^\bullet \rightarrow D^\bullet \ldots $$ of quasiisomorphisms (or weak equivalences) of dglas in order to "compute" germs of deformation spaces. This is exactly what Goldman and Millson did in their paper (with a followup by Simpson) in order to prove that singularities of representation varieties of Kaehler groups are quadratic.
The intuitive reason for the Equivalence Theorem is that $H^1$ describes infinitesimal deformations while $H^2$ describes obstructions to integrability of all orders for the infinitesimal deformations (both observations go back, I think, to the work of Kodaira and Spencer). In particular, if infinitesimal deformations and all obstructions match then one gets an analytical isomorphism of germs. The isomorphism is merely analytical since one has to deal with infinite power series, as one has to kill obstructions of infinitely many orders. Occasionally, after an analytical reparameterization of the germ, there is only one obstruction (the first one), as in the theorem of Goldman and Millson (in this case, the deformation problem is formal), but this is a bit of a miracle, coming from Kaehler geometry.
Addendum: After looking at this paper by Lurie linked in David's answer to another question (see David's comment), I see that all deformation problems, at least on the level of formal neighborhoods, are determined by some dglas. This is something I did not realize before; I guess, I never think about mathematics in this degree of abstraction. I did find it surprising though, that Lurie did not mention Deligne's letter and the paper by Goldman and Millson, maybe representation varieties do not appear among the deformation problems he was thinking about.

3$\begingroup$ This is a very nice answer. An over concise summary (as in this question mathoverflow.net/questions/385/…) is that in (the appropriate refinement of) the derived category any deformation theory becomes simply the study of the inverse processes of taking tangents (leading to dgla's as above) and exponentiating (passing to formal deformation spaces). $\endgroup$ Commented Nov 3, 2012 at 16:29

$\begingroup$ I agree that GoldmanMillson's theory and Deligne's letter, as discussed in Millson's ICM address (together with important insights of Drinfeld, Feigin and Kontsevich) are certainly among the crucial historical origins of the current understanding of deformation theory. In some sense representation varieties are behind what I might consider the origin of this entire line of reasoning, in Quillen's amazing work on rational homotopy theory and the cotangent complex.. $\endgroup$ Commented Nov 3, 2012 at 18:19