What is the cotangent complex good for? The cotangent complex seems to be a pretty fundamental object in algebraic geometry, but if it's treated in Hartshorne then I missed it. It seems to be even more important in derived algebraic geometry, so I think I need to slow down and zoom out a bit. When first learning about object $X$, it's nice to have in mind some concrete applications of $X$ to structure one's thinking.
Question: Why study the cotangent complex? What problems is it intended to solve?
(Bonus points if there is something interesting to say about extending to the derived setting.)
I have the sense that the cotangent complex is such a fundamental object that it may be difficult to isolate its importance -- much like trying to articulate the significance of something like cohomology. In that case, it might be more appropriate to ask something like "what kinds of questions does the cotangent complex allow one to ask?".
EDIT: The answers so far are great, but I imagine there are a great many more examples which could be given (the more down-to-earth the better!). As suggested in the comments, it's probably appropriate to say a bit more about where I'm coming from.
I suppose the main ideas I have in my head right now are:

*

*The cotangent complex generalizes the Kahler differentials.


*The cotangent complex controls deformation theory.
This leaves me with a few difficulties:

*

*I'm not used to thinking of differential forms primarily as "things that control deformations". So it might be helpful to simply illustrate the use of the cotangent complex by describing some deformation problem and its solution in the smooth case using differntials -- it would then seem natural to want to generalize this situation to the non-smooth case.


*I'm not even sure why I should be interested in deformation theory as such. So it might be helpful to simply see an example of a problem which arises outside the context of deformation theory itself, see how it can be rephrased deformation-theoretically, and then see how its solution uses the cotangent complex. Bonus points if the story is geometric enough to see why the role of the cotangent complex here is really a generalization of the role of differentials.
 A: This is not down a stacky and geometric alley many people like, but I think it maybe useful to show how the (algebraic) cotangent complex can be used to organize and solve a very natural question of comparing cotangent cohomology theories as the classical HKR theorem does.
In joint work with R. Campos, we managed to use the cotangent complex (which is defined functorially for algebras over say algebraic operads) to solve the following problem:

Suppose that $f:P\longrightarrow Q$ is a morphism of (algebraic) operads and consider the induced morphism on cotangent complexes $\mathbb L_{P,A} \longrightarrow \mathbb L_{Q,A}$. If $A$ is a smooth $Q$-algebra (meaning the functor $X\to \operatorname{Der}_Q(X,M)$ is exact),
when can we find a functor $F$ that produces an HKR-type quasi-isomorphism of complexes
$ \mathbb L_{P,A} \longrightarrow F(\mathbb L_{Q,A})?$

It turns out that if one considers the category of left dg-$P$-modules, then such functors $F$ are more-or-less in correspondence with resolutions of $Q$ as a left $P$-module through $f$,  and we managed to show that

If $f$ is left Koszul (meaning, we can choose a left resolution $(P\circ F,d)$ that is diagonally pure) then the generators $F$ of the resolution solve the problem above: for every smooth $Q$-algebra $A$ there is a map of complexes $\mathbb L_{P,A} \longrightarrow F(\mathbb L_{Q,A})$ that is a quasi-isomorphism.

To see how this is an HKR theorem, observe that for the projection $A\to C$ of the associative operad onto the commutative one, we can find a diagonally pure resolution given by $(A\circ F,d)$  where $F =\mathsf{Lie}^¡$ has weight degree equal to arity. Indeed, you get that the homology of this is $C$ because $A = C\circ \mathsf{Lie} $ and $\mathsf{Lie} \circ \mathsf{Lie}^¡ \simeq k$.
This recovers the classical HKR theorem, since of course $F(V) = S^c(V)$ and so we get that $HH_*(A)$ is the free cocommutative algebra over $\Omega_A^1$ (since $A$ is smooth, this is all that is left in homology of $\mathbb L_{C,A}$).
A: Let me try to answer the questions in your edit.  For the relationship to deformation theory, it is geometrically simpler to think about the tangent complex.
Let $X$ be a smooth algebraic variety.  Then  $H^0(T_X)$, the space of global vector fields on $X$, governs infinitesimal automorphisms of $X$.  This is just the usual relationship between vector fields and one parameter subgroups from differential topology. From this fact, we can see why $H^1(T_X)$ governs deformations.
Algebraically, write $D = {\rm Spec}~ k[\epsilon]/\epsilon^2$.  A deformation of $X$ is a (flat) variety $\tilde X \to D$  together with an isomorphism  $X \to \tilde X \times_D {*}$.  Deformations of $X$ form a groupoid in the obvious way.  Further, for every open subset $U$ of $X$ we can consider the groupoid of deformations of $U$.  This gives us a sheaf of groupoids on $X$,  $U \mapsto Def(U)$.
Now the key facts are:

*

*If $U$ is affine, then every deformation of $U$ is isomorphic to $U \times D$.

*The automorphism group of the deformation $U \times D$ is canonically isomorphic to $H^0(T_U)$.

The second fact is an algebraic restatement of the fact from before.  The first one uses smoothness of $X$.  Granting these two facts,  it follows formally that the sheaf of groupoids is the one associated to the sheaf of groups $H^0(T_U)$. So in particular the set of connected components is $H^1(T_X)$.
Now, since we are working in a homological/homotopical setup, it is intuitive that to carry out a similar story for a singular scheme, we should resolve it by smooth schemes, and work from there.  In other words, if $X$ is singular, then the automorphism group of $X \times D$ is not necessarily governed by $H^0(T_X)$ any more, but we know we can compute its automorphism group in the category of derived schemes by resolving $X$ by smooth schemes.  You can try to think about what happens with the simplest singularity $\mathbb V(xy)$.
As for applications of deformation theory-- algebraic geometers want to classify varieties.  Moduli spaces/stacks are very useful, and deformation theory precisely tells you the local structure of moduli space.  For instance,  the moduli space of genus $g$ curves is smooth and $3g-3$ dimenisonal.  Why? Because $H^1(T_C) = H^0(\Omega_C^{\otimes 2})$ has dimension $3g-3$ always by Riemann Roch.  The cotangent complex comes in when you want to compactify the moduli-space, the boundary of the compactification will consist of singular curves.  You can look at Deligne-Mumford's paper to see how deformation theory gets used here.
A: Here is an example mentioned in passing by user ali's answer, but I think it is cute (and powerful) enough to be worth fleshing out the details.
Lifting from characteristic $p$ to characteristic zero
In short, studying a geometric object (say, a scheme) $X$ in characteristic $p$ often involves lifting it to characteristic zero. For example, if $X$ is a smooth projective variety over $\mathbf{F}_p$, we may try to find a (flat) lift $\mathcal{X}$ over the $p$-adic numbers $\mathbf{Z}_p$. Now, $\mathbf{Z}_p$ embeds into $\mathbf{C}$ (in some completely noncanonical way), and we can apply powerful methods such as Hodge theory to the complex manifold underlying $\mathcal{X}_\mathbf{C}$.
Now, recall that
$$ \mathbf{Z}_p = \varprojlim_n \mathbf{Z}/p^{n+1}. $$
Thus lifting $X_0=X$ over $\mathbf{Z}_p$ involves finding compatible liftings $X_n$ over $\mathbf{Z}/p^{n+1}$ for all $n$. The system $\mathfrak{X} = \{X_n\}$ (or its inductive limit in locally ringed spaces) is a "$p$-adic formal scheme," and the next step involves checking that it is algebraizable, i.e. that it comes from an actual scheme $\mathcal{X}/\mathbf{Z}_p$ by the obvious "formal completion" functor.
Now the first step, finding the successive liftings $\{X_n\}$, is completely controlled by deformation theory. In our situation, it says the following:

*

*If $X_0$ is a scheme over $\mathbf{F}_p$, and $X_n$ is a flat lifting of $X_0$ over $\mathbf{Z}/p^{n+1}$, there there exists an obstruction class
$$ {\rm obs}(X_n, \mathbf{Z}/p^{n+2}) \in {\rm Ext}^2(\mathbf{L}_{X_0/\mathbf{F}_p}, \mathcal{O}_{X_0}) = {\rm Hom}_{D(X_0)}(\mathbf{L}_{X_0/\mathbf{F}_p}, \mathcal{O}_{X_0}[2]), $$
which vanishes if and only if there exists a flat lifting $X_{n+1}$ of $X_n$ over $\mathbf{Z}/p^{n+2}$. It is functorial in the sense that for $f_n\colon X_n\to Y_n$ lifting $f_0\colon X_0\to Y_0$ we have a commutative square
$$\require{AMScd} \begin{CD}
\mathbf{L}_{Y_0/\mathbf{F}_p} @>>> \mathcal{O}_{Y_0}[2]\\ @VVV @VVV\\
Rf_{0, *}\mathbf{L}_{X_0/\mathbf{F}_p} @>>> Rf_{0, *}\mathcal{O}_{X_0}[2]
\end{CD}$$


*In case the obstruction class vanishes, the set of isomorphism classes of such liftings $X_{n+1}$ is in a natural way a torsor under
$$ {\rm Ext}^1(\mathbf{L}_{X_0/\mathbf{F}_p}, \mathcal{O}_{X_0}) = {\rm Hom}_{D(X_0)}(\mathbf{L}_{X_0/\mathbf{F}_p}, \mathcal{O}_{X_0}[1]). $$


*The group of automorphisms of any lifting $X_{n+1}$ restricting to the identity on $X_n$ is naturally isomorphic to
$$ {\rm Hom}(\mathbf{L}_{X_0/\mathbf{F}_p}, \mathcal{O}_{X_0}). $$


*There is a similar story for lifting morphisms $f_0\colon X_0\to Y_0$.
So if you can show that ${\rm Ext}^2(\mathbf{L}_{X_0/\mathbf{F}_p}, \mathcal{O}_{X_0})$ vanishes, then you know that $X_0$ admits a formal $p$-adic lifting $\mathfrak{X}$. For example, if $X_0$ is a K3 surface, then this group can be identified with the space of global vector fields on $X_0$, and its vanishing is a difficult theorem due to Rudakov and Shafarevich. (And the fact that there is an algebraizable formal lifting, i.e. that an ample line bundle can be lifted to all $X_n$'s for a good choice of $\mathfrak{X}$, was shown later by Deligne.)
Perfect schemes and Witt vectors
Recall that for every perfect field $k$ of characteristic $p>0$ there exists a unique complete discrete valuation ring $W(k)$ (its ring of Witt vectors) with residue field $k$ whose maximal ideal is generated by $p$. It is a functor of $k$, and we have $W(k) \simeq k^{\mathbf{N}}$ as functors into sets. The addition and multiplication laws on $k^{\mathbf{N}}$ obtained this way are given by complicated universal formulas, e.g.
$$ (x_0, x_1, \ldots) + (y_0, y_1, \ldots) = (x_0 + y_0, x_1 + y_1 - \sum_{0<i<p} \frac 1 p \binom p i x_0^i y_0^{p-i}, \ldots). $$
We define $W_n(k) = W(k)/p^n$ and call these Witt vectors of length $n$.
For example, $W(\mathbf{F}_p) = \mathbf{Z}_p$, $W_n(\mathbf{F}_p) = \mathbf{Z}/p^n$.
In fact, the above can be defined for any ring $R$. If $R$ is a perfect $\mathbf{F}_p$-algebra, meaning that its Frobenius
$$ F_R \colon R\to R, \quad F_R(x) = x^p $$
is an isomorphism, then $W(R)$ is a flat lifting of $R$ over $W(\mathbf{F}_p) = \mathbf{Z}_p$.
Here is a beautiful argument (I think due to Bhargav Bhatt) employing the cotangent complex to show the existence of Witt vectors for perfect rings (or schemes) without using any strange-looking universal formulas for addition and multiplication.

Theorem. Let $X$ be a perfect $\mathbf{F}_p$-scheme. There exists a unique up to unique isomorphism formal $p$-adic lifting $\mathfrak{X} = \{X_n\}$ of $X_0=X$. Moreover, every morphism $f\colon X\to Y$ admits a unique lifting $\mathfrak{X}\to \mathfrak{Y}$.

The above implies that $\mathfrak{X}$ is a functor of $X$, denoted $W(X)$.  It is not difficult to prove that it indeed coincides with the Witt vectors.
Proof. Consider the cotangent complex $\mathbf{L}_{X_0/\mathbf{F}_p}$ and the map
$$ F_X^* \colon \mathbf{L}_{X_0/\mathbf{F}_p}\to F_{X, *} \mathbf{L}_{X_0/\mathbf{F}_p} $$
induced by the absolute Frobenius $F_X\colon X\to X$. Since $F_X$ is an isomorphism, the map $F_X^*$ is an isomorphism too. The complex $\mathbf{L}_{X_0/\mathbf{F}_p}$ is defined by locally resolving $\mathcal{O}_X$ by free $\mathbf{F}_p$-algebras and considering their Kaehler differentials. And $F_A$ acts as zero on $\Omega^1_{A/\mathbf{F}_p}$ for every $\mathbf{F}_p$-algebra $A$:
$$ F_A^*(dx) = dF_A(x) = dx^p = px^{p-1} dx = 0. $$
Therefore the map $F_X^*$ above is the zero map. Since it is also an isomorphism, we conclude that $\mathbf{L}_{X_0/\mathbf{F}_p} = 0$!
Now by deformation theory, the obstructions to lifting lie in the zero group (and hence the successive liftings exist), the isomorphism classes of different successive liftings are permuted by the zero group (and hence the liftings are unique), and their automorphism groups are trivial (so the liftings are unique up to a unique isomorphism). Similarly, one handles the lifting of morphisms. $\square$
A: One major application is in Artin's representablity theorem.  Existence of a cotangent complex (what Artin calls existence of an obstruction theory) allows you to linearize the question of the existence of a smooth atlas.  It's really powerful.  I'd really suggest looking through DAG XIV if you're interested in this application (the derived version is in DAG-0 (Lurie's thesis)) and a simplified version is in one of the appendices of HAG-II by Toën and Vezzosi.
Representability was used to stunning effect in Toën's 2011 paper showing that every derived fppf-algebraic n-stack is admits a smooth atlas and therefore is a derived Artin algebraic n-stack, and vice-versa (showing that all Artin derived n-stacks are fppf sheaves).
I recently had to make use of these theorems to prove fppf descent for étale n-sheaves on spectral DM stacks in something I've been working on recently.  The key point in all of this is showing the existence of a cotangent complex.
A: If you want to see the relation between differential forms and deformation theory you can look at the part B of Illusie article in FGA explained which shows that why the obstruction to the problem of lifting a morphism or a scheme to an infinitesimal neighborhood lies in the cohomology of a sheaf related to sheaf of differentials. It also discuss some classic application of this fact about fundamental group of schemes and similarly this is very important fact when you want to prove the base change theorems about étale cohomology.
One of the main motivation of cotangent complexes was studying deformations of $p$-divisible groups and proving Grothendieck–Messing theorem. This is important because for example you have a criterion for smoothness based on deformations which is in particular useful when you defined your scheme by it's functor of points. You can use Grothendieck–Messing theory to prove smoothness of Shimura varieties (and their integral models) among other things.
The cotangent complex is also useful in perfectoid geometry because you often want to lift a morphism or a scheme from residue field of a complete local ring and deformation theory and cotangent complex are the main tool for this kind of problems.and in the setting of prefectoid rings the cotangent complex (or at least its derived p-adic completion) vanish because of the surjection of the Frobenius. it is an easy exercise once you build the machinery of cotangent complexes but it is very important because you don't have any obstruction to lift morphisms.
