What is the inverse image sheaf necessary for in algebraic geometry? Given a continuous map $f \colon X \to Y$ of topological spaces, and a sheaf $\mathcal{F}$ on $Y$, the inverse image sheaf $f^{-1}\mathcal{F}$ on $X$ is the sheafification of the presheaf
$$U \mapsto \varinjlim_{V \supseteq f(U)} \Gamma(V, \mathcal{F}).$$
If $X$ and $Y$ happen to be ringed spaces, $f$ a morphism of ringed spaces, and $\mathcal{F}$ an $\mathcal{O}_X$-module, one then defines the pullback sheaf $f^* \mathcal{F}$ on $X$ as
$$f^{-1}\mathcal{F} \otimes_{f^{-1} \mathcal{O}_Y} \mathcal{O}_X.$$
However, I cannot think of any other usage of the inverse image sheaf in algebraic geometry.  Moreover, if $X$ and $Y$ are schemes and $\mathcal{F}$ is quasicoherent, there is an alternate way of defining $f^* \mathcal{F}$. Given $f \colon \mathrm{Spec} B \to \mathrm{Spec} A$, and $\mathcal{F} = \widetilde{M}$, where $M$ is an $A$-module, one defines $f^* \mathcal{F}$ to be the sheaf associated to the $B$-module $M \otimes_A B$.  To extend this to arbitrary schemes, it is necessary to prove that it is well-defined; but I still think it is easier to work with than the other definition, which involves direct limits and two sheafifications of presheaves (the inverse image, and the tensor product).  I have not checked, but I imagine that something similar can be done for formal schemes.
Hence, my question:

What uses, if any, does the inverse image sheaf have in algebraic geometry, other than to define the pullback sheaf?

A closely related question is

In a course on schemes, is there a good reason to define the inverse image sheaf and the pullback sheaf for ringed spaces in general, rather than simply defining the pullback of a quasicoherent sheaf by a morphism of schemes?

To go from the first question to the second question, I suppose one must also address whether there are $\mathcal{O}_X$-modules significant to algebraic geometers that are not quasicoherent.
Edit: I think the question deserves a certain amount of clarification.  Several people have given interesting descriptions or explications of the inverse image sheaf.  While I appreciate these, they are not the point of my question; I am, specifically, interested to know whether there are constructions or arguments in algebraic geometry that cannot reasonably be done without using the inverse image sheaf. So far, the answer seems to be that such things exist, but are not really within the scope of, say, a one-year first course on schemes. There are other constructions (such as the inverse image ideal sheaf) that do not, strictly speaking, require the inverse image sheaf, but for which it may be more appropriate to use the inverse image sheaf as a matter of taste.
 A: Donu Arapura (and BCnrd) already made this point, but I want to emphasize it: algebraic geometry employs a whole universe of sheaves that do not have $\mathscr{O}_X$ actions, and in those cases, the inverse image is the pullback of choice.  Standard examples include:


*

*Sheaves of solutions to a system of linear algebraic differential equations, in other words, flat sections of a quasicoherent sheaf with respect to a connection.  Sometimes these are reasonably familiar, e.g., when they are locally constant.

*$\ell$-adic sheaves on (the étale site of) a variety over a finite field of characteristic $p$ - this was the first toolset for proving the Weil conjectures.

*Sheaves of sets, for studying representability and so on.

*Sheaves of commutative monoids, in log geometry.

*Sheaves of closed differential forms (which appear when studying e.g., characteristic classes related to twisted differential operators)


I've definitely seen the inverse image employed in the first 4 cases, and I wouldn't be surprised if it appeared in the fifth.
A: I prefer the definition of $f^\*$ as a left-adjoint to $f_* : Mod(X) \to Mod(Y)$. The formula involving inverse image is then basically abstract nonsense using a transitivity argument with constant sheaves, at least philosophical. The proof of existence is another issue, but it follows from rather general facts of category theory (Kan extensions).
Anyway, your question was about the use of $f^{-1}$ in algebraic geometry. An example is the reduced structure sheaf on a closed subset of a locally ringed space. You take the vanishing ideal $I$ and then pull back $\mathcal{O}_X / I$ along the inclusion map, which is a priori just a continuous map. You can also view this as a module pull back, but only if you have already defined the structure sheaf.
Also, I think it is very important to learn the somewhat old-fashioned view on sheaves, namely as sections of the etale space. Then you quickly arrive at the question to which sheaf corresponds to restriction of the etale space to a subset, which is not necessarily open. Well, it is just the pullback with respect to the inclusion map.
Finally, it is good to know that the morphism $f^{\#} : \mathcal{O}_Y \to f_{*} \mathcal{O}_X$, appearing in the definiton of a morphism of a ringed spaces, corresponds to a morphism $f^{-1} \mathcal{O}_Y \to \mathcal{O}_X$, from which you get the stalk maps directly.
A: By some coincidence, I have a student going through this stuff now, and we got to this point this just yesterday.
The definition of $f^{-1}$ is certainly disconcerting at first, but it's not that bad.
You'd like to say
$$f^{-1}\mathcal{F}(U) = \mathcal{F}(f(U))$$
except it doesn't make sense as it stands, unless $f(U)$ is open. So we approximate
by open sets from above. A section on the left is a germ of a section of $\mathcal{F}$ defined in some
open neighbourhood of $f(U)$, where by germ I mean the equivalence class where you identify two sections if
they agree on a smaller neigbourhood.
Even if you're still unhappy with this, the adjointness property tells you that it
is the right thing to look at.
Also, some of us work with non-quasicoherent sheaves (e.g. locally constant sheaves or constructible sheaves), so it's nice to have a general construction.
Addendum: In my answer yesterday, I had somehow forgotten to mention
the etale space or sheaf as a bunch of stalks 
$$\coprod_y \mathcal{F}_y\to Y$$
viewpoint 
discussed by Emerton and  Martin Brandenburg.  Had you started with this "bundle picture",
we would be having this discussion in reverse, because pullback is the natural operation here and pushforward is the thing that seems strange.
A: Here is a fairly polemical answer, in a similar spirit to Brian's:
Sheafification is not a painful process: you take a presheaf, and you think about how you need to change it so that the stalks are the same, but sections can be glued.  It is very natural.
The inverse image is also naturally understood in the same kind of terms: you have a sheaf $\mathcal F$ on $X$, and you would like to make a sheaf on $Y$ whose stalk at $y$ is equal
to the stalk of $\mathcal F$ at $f(y)$ (i.e. $(f^{-1}\mathcal F)_x = \mathcal F\_{f(x)}$).
If you ponder how you can make a rigorous construction with these properties, you will be led
to the inverse image.  (It's essentially taking the fibre product of $\mathcal F$ over $X$
with the map $f:Y \to X$, and indeed thinking about the inverse image is good practice for developing intuitions about fibre products in lots of other contexts.)
Using the crutch of affines and quasi-coherent sheaves discourages thinking about the (fairly simple and natural) local picture of a sheaf as a bunch of stalks glued together.
A lot of the power of the geometric ideas in algebraic geometry comes from thinking geometrically, so one doesnt' want to be discouraging thinking about sheaves in this way;
rather, you want to encourage it.
As for applications, Donu notes some in his answer.  
Let me note another here: if $\mathcal I$ is an ideal sheaf on $X$, then $f^{-1}\mathcal I$
is naturally a subsheaf of $f^{-1} \mathcal O_X$ (because $f^{-1}$ is exact, as one sees immediately by looking on stalks and using the fact that $f^{-1}$ doesn't change stalks!), and one often wants to look at the ideal sheaf
in $\mathcal O_Y$ generated by this.  This is not the same (typically) as $f^*\mathcal O_Y$.  (Just as, if $I$ is an ideal in $A$ and $B$ is an $A$-algebra, $B\otimes_A I$ is typically is not isomorphic to the ideal in $B$ generated by $I$.)
Now there are other ways to describe this ideal sheaf in $\mathcal O_Y$ (e.g. it is the image
of the natural map $f^*\mathcal I \to \mathcal O_Y$), but the description of it in terms
of $f^{-1}\mathcal I$ is convenient and very natural.
A: One quick answer is that the stalk of a sheaf $F$ at a point (say, given by an inclusion $f\colon pt \to X$) is just $f^{-1}F$. 
A: Roughly speaking, an element of $\varinjlim_{V\supseteq f(U)} \Gamma(V, \mathcal{F})$ is just a section in an open neighborhood of $f(U)$ (with some proper identifications).  
On the other hand, a section $s\in \Gamma(U, f^{-1}\mathcal{F})$ is 
given by an open cover $\{V_i\}_{i\in I}$ of $f(U)$ and a section $s_i\in \Gamma(V_i, \mathcal{F})$ for each $i\in I$ for which we require that 
$$ s_i\mid_{V_i\cap V_j \cap f(U)} = s_j\mid_{V_i\cap V_j \cap f(U)}.$$
Here equality means stalk-wise equality. In other words, $s_i=s_j \in \mathcal{F}_y$ for every $y\in V_i\cap V_j \cap f(U)$.  Given two such collections $\{(s_i, V_i)\}$ and $\{(t_j, W_j)\}$, we say they are equal if they match stalk-wise for every point $y\in f(U)$. 
The difference is just that one substitute a "global" statement where a section in $\Gamma(U,f^{-1}\mathcal{F})$ is required to extend to a nbhd of the whole of $f(U)$ for a "local" version of essentially the same statement and require the sections to glue together on $f(U)$.  
For example, if we have $V_1\cup V_2 \supseteq f(U)$ and sections $s_i$ over the respective open sets. We might have $s_1$ and $s_2$ matches on $f(U)\cap V_1\cap V_2$ but differs at some point $y\not\in f(U)$, so they fail to glue to a section on $V_1\cup V_2$. 
The good thing here is that in this case the support of $s_1-s_2$ is closed in $V_1\cap V_2$, after shrinking our open sets $V_1$ and $V_2$, the sections do match on the intersection.  Of course you expect this method to fail when infinite open sets are involved.  This also leads to the idea that once compactness is involved, then one may have hope for this to work.  For example, see Lemma 1 of Akhil Mathew's Note.
