My first advice is to stop worrying about what progression you should be learning stuff. Just read things you find interesting, and if you find yourself reading something that makes no sense to you then you can start backtracking and looking at prerequisites. You don't need to plan ahead so much. Anyway, I second the recommendation of the book of Mac Lane and Moerdijk.

Also, I don't think it sounds absurd to study category theory without having a background in set theory/logic. Most people who know a lot of category theory do not, either; I would say most applications of category theory are in topology/algebraic geometry and related areas. But there are also logicians (as you mention), computer scientists, mathematical physicists, and many others that apply category theory in their work.

Finally, there's a big risk your question is going to closed. Don't take it personally, it's just a bit too discussion-y to be a great fit for the MO "format".