I see three questions.
Question 1: Why do we care about $Set$ when we define functors $C\to Somebase$?
$Set$ is a very special category:
- $Set$ is the terminal object of the category of Grothendieck topoi and geometric morphisms
- $Set$ is the free cocompletion of the one-object category.
- $Set$ is a "canonical choice" for a category whose terminal object is a generator, and where you can embody the set of natural numbers.
- the mystics claim that in $Set$ every epimorphism has a right inverse.
Question 2: why we do it for other categories as well?
A presheaf is a functor $F : C\to Set$; if $C=1$ this is a "static" set, but as soon as $C$ is bigger the set $Fc$ varies concordantly with the variability of structure in $C$. If you join this with the fact that, in a very precise sense, $F$ is a "space" ${\cal E}(F)$ lying over $C$, you get that it is a nice idea to study families of sets parametrized by a category.
Another analogy that I like very much is that presheaves describe actions of a category $C$ on sets, one set for each object of $C$, and consistently with morphisms of $C$ (what is a presheaf on a monoid? What is a functor $G \to Vect_k$ if $G$ is a group? In both cases $G$ is a one-object category).
In both these pictures it is a very natural request that the images of $c\in C$ under $F$, or equivalently the fibers of $\Pi : {\cal E}(F) \to C$ over $c$, are structured sets. Hence the need for presheaves of abelian groups, rings, $R$-modules, topological spaces, categories: in which way I can represent variable structured sets? In which ways I can make a category act on structured sets?
This intertwines with another important point, that also adresses your third question: in the internal language of a category of functors $[C,Set]$ you have the ability to speak about structured objects: you can say what a monoid, a group, a ring, a module, a poset... is in $[C,Set]$, by asking that certain maps (representing operations) make certain diagrams (representing properties of those operations) commute. The theory of internal structures is something we investigate since Lawvere's thesis (Functorial semantics), and his work, as well as the huge literature in topos theory, sheds a light on the precise meaning of the following items:
- "Structures are categories", in the sense that there exists a category $Th(Grp)$ such that suitable functors $Th(Grp)\to Set$ correspond precisely to groups as naively defined to you day 1 of your first algebra course. The same things happens, in the very same way, for every other algebraic structure, at least those who satisfy a certain technical property (the category of the functors $Th(\Omega)\to Set$ is monadic over $Set$. The category $Th(\Omega)$ contains the archetypical shape that any $\Omega$ structure shall have; it is called a "theory". A functor $Th(Grp)\to Set$, i.e.a group, is a model for that theory. But models can exist in every context expressive enough to embody them. So,
- as sets correspond to the category $[1,Set]$, it is a natural question what structures internally to more general $[C,Set]$ are (and perhaps one day to even more general finitely-complete categories $K$). In a stunning turn of events, now, (say) an abelian group, i.e. a functor $Th(AbGrp)\to [C,Set]$ is precisely a functor $C\to Set$ such that each $Fc$ is an abelian group, i.e. the "models" for the "theory" of abelian groups internally to $[C,Set]$ are precisely the functors $C\to Set$ taking values in the subcategory of models for the theory of abelian groups in $Set$. If we call $Mod_{Th(\Omega)}( K)$ the $\Omega$-structures internal to the category $K$ we can "shift" the $Mod(-)$ correspondence in and out $[C,Set]$:
$$
Mod_{Th(AbGrp)}([C,Set]) \cong [C, Mod_{Th(AbGrp)}(Set)]
$$
This seemingly trivial statement fills me with wonder every time I see it.
So, somehow, once you are able to classify models for $\Omega$-structures in $Set =[1,Set]$, you are automatically able to classify models in every other $[C,Set]$. So sets, in this respect, play a central rôle!
This brings me to your last question, maybe the most intriguing one of the list.
Question 3: Why it seems that sheaves contain interesting information?
Long story short: because they do.
The internalization paradigm sketched above shows that "small" mathematicians could live in a single big, finitely complete category $K$, without even worrying about the presence of models for their theories outside it.
Thus, if you admit them to be big enough (i.e. if you leave the somewhat unsatisfying picture that "all categories are small"), each category works as a universe in which you can speak mathematical language ("speak the language of mathematics" is synonym, since model theory, with "studying models for the theory of $\Omega$-structures" as long as $\Omega$ runs over all possible theories). Small mathematicians are born in $K$, so they only see $K$-models for $Th(\Omega)$. (some small mathematicians even have problems accepting that what they're used to call "abstract groups", are instead models for an even-more-abstract theory of groups).
As an aside and flamy comment, (i) the fact that until your mathematics is small, you can ignore the existence of large objects, and (ii) the difficulty to exit $K$, explains at least to a certain extent the resistance to categorical thinking.
To remedy the apparently arrogant tone of this last remark, let me stress that "small" has no pejorative meaning whatsoever, in the same sense a small category is not less legitimate to exist.
So categories are universes in which you can interpret theories, and if you look from high enough there is plenty of other objects having the same properties of $Set$, and you shouldn't be afraid to move there (or rather, you shouldn't insist on living in $Set$: maybe your destiny is to conquer the category of sheaves on $S^n$, or the category of functions whose codomain is $\mathbb N$).
This apparently philosophical discussion opens instead a rather profound point: each category $K$ has a semantics, suited to interpret theories in $K$.
To be more precise, then, categories are places, in each of which we can interpret different kinds of logics (if you like categorical thinking, you better familiarize with the idea that there is plenty of different logics, in the same way -and with the same continuous spectrum of nuances in flavour and aroma- that there is plenty of different coffee beans; also, more than often the debate about the best coffee leads to the same war of religion the debate about the best logic does).
As you can see on the nLab page about internal logics, what determines the particular shape of semantics that you can interpret in $K$ is no more, no less than the nice categorical properties of $K$ (does it have finite co/limits? Does it have a nice factorization system? Does it have a subobject classifier? Is the poset $Sub(A)$ of subobjects of $A$ a lattice, is it modular, distributive, complemented...? -evidently this last question is about the internal logic of the category: propositions "are" the set, or rather the type, of "elements" for which the proposition is "true").
Categories of sheaves are nice objects in this respect because they have almost all these properties: subobjects can be joined, united, complemented; finite co/limits, and even infinite ones, exist. Subobjects are realized by maps to a "classifying" object $\Omega$, whose "elements" are truth values of propositions; each slice category $Sh(C)/P$ has the same properties and the functors $Sh(C)/P \leftrightarrows Sh(C)/Q$ induced by maps $P\to Q$ are "nice" (they preserve this structure).
A category of sheaves is then a good choice for the $K$ in which you want to live, simply because it is expressive enough to do a lot of mathematics.
What I'm leaving outside of this comment, since I've alrady ridiculed myself in front of people who actually work with topoi and know this topic better than I do, is the following:
- mention accessible and locally presentable categories. Categories of sheaves are locally presentable elementary topoi. These are very nice categories that albeit being large can be described by a set (sometimes a group is infinite, but can be generated by a finite set; sometimes a category is a proper class, but there is a set of objects generating it).
- Work "on a relative base": sets are no more special than another topos, they're only easier to handle. Most of topos theory works the same way if you fix a "base" topos once and for all, say $\cal S$, and study the 2-category of "topoi over $\cal S$".
- Since a topos is a generalized space, it turns out that this procedure is no different from studying the category of space-maps over a prescribed base $X$ as opposed to the category of all maps with variable codomain. $Set$ being the terminal object of $\bf Topoi$, ${\bf Topoi}/Set$ is merely $\bf Topoi$.
