Categories First Or Categories Last In Basic Algebra?  Recently, I was reminded in Melvyn Nathason's first year graduate algebra course of a debate I've been having both within myself and externally for some time. For better or worse, the course most students first use and learn extensive category theory and arrow chasing is in an advanced algebra course, either an honors undergraduate abstract algebra course or a first-year graduate algebra course.
(Ok, that's not entirely true, you can first learn about it also in topology. But it's really in algebra where it has the biggest impact. Topology can be done entirely without it wherareas algebra without it beyond the basics becomes rather cumbersome. Also, homological methods become pretty much impossible.) 
I've never really been comfortable with category theory. It's always seemed to me that giving up elements and dealing with objects that are knowable only up to isomorphism was a huge leap of faith that modern mathematics should be beyond. But I've tried to be a good mathematican and learn it for my own good. The fact I'm deeply interested in algebra makes this more of a priority. 
My question is whether or not category theory really should be introduced from jump in a serious algebra course. Professor Nathanson remarked in lecture that he recently saw his old friend Hyman Bass, and they discussed the teaching of algebra with and without category theory. Both had learned algebra in thier student days from van der Waerden (which incidently, is the main reference for the course and still his favorite algebra book despite being hopelessly outdated). Melvyn gave a categorical construction of the Fundamental Isomorphism Theorum of Abelian Groups after Bass gave a classical statement of the result. Bass said, "It's the same result expressed in 2 different languages. It really doesn't matter if we use the high-tech approach or not." Would algebracists of later generations agree with Professor Bass?  
A number of my fellow graduate students think set theory should be abandoned altogether and thrown in the same bin with Newtonian infinitesimals (nonstandard constructions not withstanding) and think all students should learn category theory before learning anything else. Personally, I think category theory would be utterly mysterious to students without a considerable stock of examples to draw from. Categories and universal properties are vast generalizations of huge numbers of not only concrete examples,but certain theorums as well. As such, I believe it's much better learned after gaining a considerable fascility with mathematics-after at the very least, undergraduate courses in topology and algebra. 
Paolo Aluffi's wonderful book Algebra:Chapter 0, is usually used by the opposition as a counterexample, as it uses category theory heavily from the beginning. However, I point out that Aluffi himself clearly states this is intended as a course for advanced students and he strongly advises some background in algebra first. I like the book immensely, but I agree. 
What does the board think of this question? Categories early or categories late in student training?  
 A: This is an extension to part of my comment in another answer. I learnt group theory and enjoyed the initial parts but then we had Sylow theory and it looked mysterious and somewhat frightening, as no real motivation in terms of earlier material was given. If students do not see the need for a piece of mathematics, (internally within the subject or for 'applications') it becomes mysterious. Category theory is not that different from group theory in this, so don't make a fuss about it.  When the material in an algebra course is simplified by doing it categorically use a bit of categorical language, don't make a fuss about it (I agree with the other answers on this.)
On the linked courses in Knots and Surfaces and on combinatorial group theory, we used categorical properties of products, as motivation for the product topology, used van Kampen theorem situations as motivation for 'free products with amalgamation' and pointed out the similarity of the observable pushout property with that of union, so the vKT is naturally about the preservation of some kind on mathematical structure namely pushout, by some kind of construction, oh dear, the concept of functor is just asking to be introduced and so on.
That was at Undergrad year 3 and MMath 4 level (in UK terms), then at Masters level, more examples led to a need to formalise things so as to make it clearer what was going on.  I think it worked and was enjoyed and understood by students. 
A: There's a big difference between teaching category theory and merely paying attention to the things that category theory clarifies (like the difference between direct products and direct sums).  In my opinion, the latter should be done early (and late, and at all other times); there's no reason for intentional sloppiness.  On the other hand, teaching category theory is better done after the students have been exposed to some of the relevant examples.  
Many years ago, I taught a course on category theory, and in my opinion it was a failure.  Many of the students had not previously seen the examples I wanted to use.  One of the beauties of category theory is that it unifies many different-looking concepts; for example, left adjoints of forgetful functors include free groups, universal enveloping algebras, Stone-Cech compactifications, abelianizations of groups, and many more.   But the beauty is hard to convey when, in addition to explaining the notion of adjoint, one must also explain each (or at least several) of these special cases.  So I think category theory should be taught at the stage where students have already seen enough special cases of its concepts to appreciate their unification.  Without the examples, category theory can look terribly unmotivated and unintuitive.
A: A little preliminary: I'm an undergraduate student and I started to study category theory as self-taught at the beginning of second year of university, mostly because of my interest in logic and foundations. Since then I've enjoyed of this fact because knowing some category theory helped me to understand lots of concepts that I've learned more quickly then what I would have done without it, also category theory move me to study some branch of maths like algebraic topology and algebraic geometry.
Now I would distinguish between "category theory" and "the language and instrument of category theory": while the first is an abstract and too specific branch of math, so not adequate to be considered in a undergraduate courses, the second is the very useful conceptual tool that should be taught also to undergraduate students.
What I mean here is that (the language of) category theory shouldn't be teached in a specific course but it should be taught during the regular courses.
I believe that some basic concepts like the ones of category and functor could be taught since first courses of algebra, that's because these concepts are not more abstract than those of groups-group homomorphism,ring-ring homomorphism, vector space-linear map which are taught in the first year's courses. Categories and functors can be easily shown to a young public respectively as graphs with structure (i.e. operations) and as graph morphisms preserving the structure. Many example can be given to those concepts which can be understood by undergraduates: the categories of graphs' points and graphs' paths, the category of sets and functions, the category of groups and group homomorphisms, vectorial spaces and linear maps, but also monoids, groups and poset as categories. 
In particular its very useful made these last example in first courses because they help in familiarizing with abstraction before mind is corrupted by concrete (I remember that after having done some basic algebra I found a lot of difficulties to understand why monoids should be categories with one object).
Another good set of examples of category the are quite easy to understand and (in my personal opinion cool) are those of objects (which can be molecules, automaton's states, dynamic system states,...) and processes transforming one object into another. These examples are pretty cool because they open the way to application of category theory also to other science, besides giving really concrete examples of categories.
Obviously categorical concepts should be introduced in a very gradual way, for instance its useless teaching natural transformation before having seen homotopies and groups' representation (or equivalently groups' actions), same apply for other more complex concepts: every thing need to be introduced at right time.
Many would object that probably concepts should be presented every time when they are needed. To those people I would say that probably they right, anyway no-one have ever introduced to me abstract concepts like the ones of groups and rings with some motivation, same apply to topological spaces, the motivations for introducing these objects came late, when where introduced some results which gives us a more abstract framework in which some kind of problems tend to simplify and generalize. 
Last motivations of teaching category theory early is that many times seeing thing from an abstract point of view helps when we want to switch constructions from categories, where these constructions are build naturally, to other categories (it comes to my mind the example of homotopies of complexes in homological algebra) and also shows deep unity of lots of mathematical objects that maybe at first seem unrelated. 
Before to end I would also like to add some motivation to why not waiting to teach category theory in advanced courses: if you do so usually happens that these categorical concepts are presented in very fast way that make difficult to take familiarity with said concepts and that doesn't allow to deeply understand the meaning and usefulness of categorical results.
One last comment: I don't know why but every time I think to those people which consider category theory too abstract and useless they remind me of what Kronecker said about Cantor, and this make me smile.
A: I wasn't going to weigh in on this as I think that this is very definitely "subjective and argumentative" (particularly the later), and when before I spoke up in favour of category theory in undergraduate education, it sparked a few comments and I was reminded of why I like the fact that discussion is suppressed in MO.  But given that one side of the argument is already here, and the other is not so well represented, I'm going to answer.
Let me start by declaring: "I am not a category theorist".  I am a differential topologist.  Foundational questions leave me cold, size issues just don't bother me.  I'll accept any axiomatic framework if someone wants me to (I'm a fully-paid-up member of the "Axiom of Choice" party).  To enter Greg's culinary world for a moment, such things are bit like Norwegian cheese.  I can see that to the right person, it's delicious.  But I'm not that person.
To continue the analogy, category theory isn't an ingredient that can be added for Extra Flavour, but which not everyone likes.  Category theory is like cooking with freshly harvested, organic ingredients as opposed to dull, insipid, shrink-wrapped stuff from the vast conglomerate supermarket.  Just making one ingredient organic doesn't have much effect on the flavour of the whole dish, but changing the whole lot does.
But to the matter in hand: undergraduates and category theory.  I believe that category theory is an excellent way to understand and express mathematical concepts.  I find in my own work that, time and time again, when I express my ideas using categorical language then it makes them clearer both to me and to others.  Believing this, as I do, why on earth would I want to deprive my students of the same benefits?
So I teach my students category theory.  I don't necessarily tell them that I'm teaching them category theory, any more than I tell them that I'm teaching them logic, or how to write proofs, or even the basics of English grammar!  But I use the insights and expressions of category theory because I think it makes it easier for the students to learn "other" mathematics.
In particular, in my current course, I am trying to teach my students the following things:


*

*To focus on processes rather than things.  Call them "morphisms" and "objects" and that's category theory!  I don't tell them to do this because that's what category theory tells us to do, I tell them to do this because that's what the Real World(TM) tells us to do: mathematics (I tell them) is about modelling the real world, and the basic thing that one wants to model is a process.

*To transfer knowledge from a known space to an unknown space.  Here we have the extension of the mathematical idea of "function over form".  That is, a thing is not defined by what it is (object) but what it does (what category it is in).  But we can take this one step further and say that it's not just what it itself does that matters, but how it relates to the things around it (what morphisms are there from it to other objects in the category?).  In particular, if I have an unknown vector space $V$ (unknown in the sense that I don't know much about it rather than I don't know how to define it), I gain a lot of knowledge if I can find an isomorphism $V \cong \mathbb{R}^n$ because I already know a lot about $\mathbb{R}^n$.
In a recent colloquium, I made this point (rather strongly) by saying that category theory is ubuntu mathematics: "I am what I am because of who we all are."

*To be able to change ones point of view to suit the problem at hand.  Say, "to look for what is preserved under isomorphism" and you've got one of the central tenets of category theory: that isomorphic objects should not be distinguished.  This is a natural extension of the above.  Once we know that an isomorphism $V \cong \mathbb{R}^n$ is a Good Thing, the next question is whether or not there's a best isomorphism (for the problem at hand).
To sum up, category theory isn't a "bit on the side" of mathematics to be taught as an optional extra at the higher levels, alongside homological algebra, Lie theory, and whatever-it-is-those-statisticians-down-the-hall-do.  It can (and should) pervade all of our teaching because it makes the learning easier.  Teaching it as a separate subject itself isn't a necessarily a bad thing, but it is if that is the only way in which it is taught, and by itself it can seem very dry, abstract, and disconnected.  But then teaching it by itself is a bit like teaching logic without ever once mentioning Raymond Smullyan.  Indeed, the comparison with logic is apt: we expect our students to pick up the basics of logic as they go along.  Not many students actually study logic as a subject by itself, but if someone asked "Should we use logic when teaching undergraduates?" it would be closed instantly as "Not a real question.".
A: In my opinion, category theory is to mathematics what garlic is to cooking.  It is a widespread ingredient that adds a very important flavor.  But, usually, it should be minced and mixed in, and used with restraint.
(So, my answer to your question of early vs late is, a little of both.)
A: Categorical ideas should be certainly introduced early, as they are quite useful. On the other hand, as Andreas and Terry say, studying category theory at the beginning of your mathematical education is a waste of time, and could be a turnoff, like all unmotivated formalism.
On the other hand, the formal language of category theory should be learned, and used, at some point. I have seen several interesting papers written by very good mathematicians, containing theorems with statements like "It is the same to give a regular thingamabob over $X$, and a von Neuman whatchamacallit with a seminormal connection over $X'$". What these statements usually mean, is that there is an equivalence of the category of regular thingamabobs over $X$ and that of von Neuman whatchamacallits with a seminormal connection over $X'$; but they could also simply mean that there is a bijection of isomorphism classes, and to know which is true you have to study the proof. This means, I suppose, that the authors, who must have seen the language of category theory at a certain point, have not interiorized it, and don't have a feeling for when its use is appropriate.
In my opinion, the concept of equivalence of categories is a real turning point. Up to that point, one can probably get away without it (for example, universal properties, like that for the tensor product, are easily explained without the formal language); this is harder to do with equivalences. On the other hand, you don't see many examples of equivalences in the beginning of your mathematical studies. Maybe the first one is that between coverings of a space, with appropriate hypothesis, and sets on which its fundamental group acts. Stating the connection between these two classes of objects as an equivalence of categories clarifies things enormously; I wish someone had explained it to me when I was a student, instead of just telling me that there is a bijection between isomorphism classes of connected covers and conjugacy classes of subgroups, and other statements in this spirit, all descending very immediately from the "real" theorem, which is the existence of the equivalence.
A: Everyone will agree with me that there are many levels of abstraction category theory can be introduced at. It makes no sense to start undergraduate math courses with a formal approach to category theory, I don't think anyone would argue the opposite. It makes very little sense either to postpone it to higher algebra classes of late undergrad at best or, as happens in many places, in graduate studies. 
Category theory is above all a formalism, a way to frame our understanding. It has been a more and more prevalent facet of my thinking that a good notation does half the work of solving a problem, just as  formulating a question properly does. Why then not start hinting towards such formulation early ? While teaching low level courses, I always have, or make a point to ensure that most of my class knows what a function is. While doing so I draw little blobs representing sets and big arrows representing a function. Then as I talk I keep presenting functions as a processes, or relations. Together with a fun example (I usually use a "friends and beer" variation) it helps them structure the knowledge they are presented with. It makes it easier to have them understand that one cannot just "add" functions by writing a plus sign in between since functions are (visually) not the same entity as numbers. It is I believe our duty to frame things as early as possible in a way that structures knowledge in the student's mind. To make another reference to food, it is better to have widely spread malleable foundations of rudimental cooking than of an elite of highly qualified cooks (of course it's best to have both).
Moreover I would like to point out that this formalism is urgently needed in other areas of science. As a physicist by training I cannot overstate the importance of category theory in  areas of science other than mathematics. And even after a MS in theoretical and mathematical physics, "functors" and "categories" were frightening words that were reserved to Jedi Masters. I am but saddened by that state of things. About everything in physics deals with processes and change and yet there seems to be very little push to spread the categorical lingua. Relativity screams category theory (equivalent views of the world in different frames yet non identical), the standard model's soul is categorical (groups, tensor structures of representations, etc...). Why should we wait so long to plant these seeds ? Why not let them germ throughout the student's curricula.
In conclusion while it is dysfunctional to force feed students categories (why teach an intensive Japanese course to someone that just wants to make suchi ?), it is criminal to keep it, to its core, our little secret. I believe we need to join forces to move very basic categorical formalism to bigger circles, sans tambours ni trompettes (without fanfare), and without bells and whistles.
A: I think that there are certain notions that are needed to "set the stage" for category theory.  I don't think students are going to understand category theory unless they've already seen some of the following examples:


*

*Galois Theory

*Covering spaces and pi_1

*The universal property of tensor product

*The difference between direct sum and direct product


If people have a very strong undergraduate background, then it seems to me that in graduate algebra they're ready to start seeing some category theory.  On the other hand, I feel like (contrary to what AndrewL said) that category theory really finds its home in topology more than algebra, so I think an algebra course should introduce category theoretic language but that it should not be the main emphasis of the course (since people who haven't taken algebraic topology probably won't really grok the category theory anyway).
A: Introductory algebra courses tend to systematically confuse products with coproducts, and more generally, confuse targets with domains.  This systematically causes confusion in students (what is the difference between the two kinds of infinite product? and why are there two kinds?  and how do I decide which to use when?).
Even if no category theory is going to be introduced, this terrible confusion should be eliminated.  
In a related note, I regard it as an extremely important point, that should be celebrated, that for abelian groups, or vector spaces, etc., sums and products agree.  In my experience, it is glossed over:  "the sum and the product are the same, so don't worry about it;  and I'll use the two notations interchangeably."
And another thing:  the free product of two groups -- really?
A: In answer to Andrew's question, I think it really depends on the student. I began learning category theory in my late teens because of the sorts of questions I was asking myself which, I discovered, could be answered through category theory. It just really "clicked" for me, and provided me with tools that I use every day in my mathematical life. 
Sometimes I would find an application of category theory to an area I didn't know too much about, but because the application seemed pretty cool, I would be motivated to learn more about the area. My own feeling is that the category theory helped me learn mathematics more quickly than I otherwise might, in part because it helped provide broad conceptual frameworks in which to fit newly acquired knowledge. So in that respect, I am happy that I began learning category theory early on. 
But category theory doesn't come naturally to a lot of people (some of the people who have answered or commented above, including some very distinguished mathematicians, don't strike me as having a whole lot of feeling for the subject). That's fine. If category theory does not come naturally to you, then simply learn category theory on a need-to-know basis, and try not to make up your mind what the subject is about (e.g., "doing away with elements") in advance. My advice is: don't force yourself to learn it unless you have a need to know (and my guess is you probably will, in tandem with other subjects). 
Over time, while studying something that you've really latched on to, you may find some categorical reasoning coming into play, and marvel at how clean and efficient it is, and how it clears away conceptual clutter. Then you may be in a proper frame of mind to make a deeper study of what makes some aspect of category theory "tick", with some heightened appreciation of what category theory is good for, or how it can serve your ends. 
A: The answers of Greg Kuperberg and Andreas Blass are great. Let me just add something:
Most of my mathematical ideas are formulated in terms of category theory. However, this does not mean that the ideas do not exist without category theory. Rather category theory is a universal language and toolbox to gather and transport these ideas.
I'm a little upset when in all these basic algebra lectures category theory is presented as something exotic and very complicated (and is postponed so later chapters). In particular when students then think that the compatibility of, say, localizations with localizations has to be checked with horrible double fractions. Perhaps this is the best moment to show them that writing down the hom functor and using Yoneda lemma simplifies the proof a lot. Then they probably appreciate this method also in other situations and start to think in morphisms rather than elements. Also, they can separate the trivial assertions from the interesting ones ;). For example it should be obvious after such a course that localizations commute with direct sums, but not necessarily with direct products (wrong arrow direction!).
But this step into category theory can only be done when there is a specific motivation. For example it is a good idea to introduce functors in a course on algebraic topology after having checked that singular homology is, what is then called, a generalized homology theory. In basic algebra categories should be introduced first when category theoretic theorems may be established without much effort which may be applied to concrete problems. For example you should not use Freyd's representabilty theorem in order to prove the existence of tensor products, when you just want to gather some basic facts about them. But it maybe a good idea to show that the equation $Hom(M \otimes N,P) = Hom(M,Hom(N,P))$ formally implies that $M \otimes -$ is right exact, and then introducing adjoint functors and other examples. Quite a few algebra texts prove the right exactness by a tedious calculation which is only a special instance of a composition of Yoneda and adjunction.
Although I don't like repetition in basic algebra courses, it makes it possible to develope some meta-theorems by analogy for yourself (which maybe will be made precise when you study advanced category theory later). For example when you have understood the construction of free abelian groups, free algebras, and other free constructions, you probably also might guess that there must be free groups, including how they are constructed and which universal property characterizes them.
