One of the best places I've found for the stuff (and it does tend to apply to the applications of categories to logic, but there's enough other stuff that it's worthwhile) is Lawvere and Rosebrugh 's "Sets for Mathematics." It does assume that you're ok with sets, and then does lots of categorical constructions with sets, and attempts to study the category of sets itself, where you have some intuition and know how to construct things (like products, coproducts, equalizers, etc). Now, it is intending to get to topoi, though most of the book is fine for just getting the categorical language, and applying it to sets.