I have plenty to say, so to simplify matters, I'll use the term "subgroup-defining function" for an isomorphism-invariant mapping that associates a unique subgroup to each group. [NOTE: It is not really a "function" because the "domain" and "range" are not sets.]
Any subgroup-defining function yields a characteristic subgroup, because the isomorphism-invariance implies invariance under automorphisms. However, it is not usually the case that a subgroup-defining function gives an endofunctor (in the sense that Scott describes), i.e., it is not necessary that it obey covariance with arbitrary group homomorphisms.
There are a few special examples of endofunctors, notably the commutator subgroup, members of the derived series, members of the lower central series, and other verbal definitions (that give verbal subgroups). In particular, if a subgroup-defining function is an endofunctor, it should, as Scott mentions, output a fully invariant subgroup, which is a weaker condition than being verbal but still stronger than being characteristic.
In fact, most subgroup-defining functions of interest are not endofunctors, and this might be related to the fact that category-theoretic, or functorial, thinking, does not come up much in the study of groups. Subgroup-defining functions such as the center, member of the upper central series, Frattini subgroup, Fitting subgroup, socle, are not endofunctors. Moreover, the subgroup they yield are not necessarily fully invariant. Interestingly, many of these subgroups are still strictly characteristic subgroups (also called distinguished subgroups) -- they are sent to within themselves by surjective endomorphisms of the group. However, this is not true for all of them.
The next question might be: is there some sense in which the subgroups obtained through easy-to-write subgroup-defining functions are more special than arbitrary characteristic subgroups? There are two related ideas that we can borrow from logic to define notions of "uniqueness" stronger than characteristicity:
There should be no other subgroup of the group such that the theories of the two group-subgroup pairs are "equivalent" in whatever logic we are working with. In first-order logic, there should be no other subgroup whose embedding in the whole group is elementarily equivalent to that of the given subgroup.
The subgroup can be defined (as a subset of the group) in the pure theory of the group, using whatever logic we are working with. In first-order logic, this is what we'd call a "purely definable subgroup" (or "definable subgroup" -- the purely is to emphasize that no additional structure has been tacked on to the group). As mentioned in Henry's reply, the center is purely definable (more generally, all members of the finite part of the upper central series are) but the commutator subgroup is not purely definable for free groups.
Even more generally, we may want that there is a pure definition that works for the subgroup-defining function, as opposed to just requiring that the output of the subgroup-defining function for each input group is definable in that group.
For each logic, (2) is stronger than (1). For instance, a purely definable subgroup cannot be elementarily equivalent to any other subgroup. Moreover, the more powerful the logic, the weaker the notions (1) and (2) in that logic.
[NOTE: That (2) in general is stronger than (1) can be seen from the analogous situation for real numbers: any two real numbers can be distinguished by a first-order predicate in the theory of real numbers, so every real number is "unique" in the sense of (1). However, there are only countably many definable real numbers, so almost all real numbers are not uniquely definable in the sense of (2).]