Well if you want to count rational points on varieties than you probably want to know what abelian varieties are, and general type varieties, and Fano varieties, and K3 surfaces, and what Azumaya algebras are, and so on to understand the main conjectures and theorems of the subject. You should probably understand how spreading out varieties into schemes over $\mathbb Z$ lets you view integral points as curves and so attack them with geometric intuition, making it clear how to properly define heights and other useful tools. This already requires quite a bit of algebraic geometry.

There may be good reasons not to learn algebraic geometry but a fear of category theory is not one of them.

First let me point out that the number of category-theoretic concepts needed is quite small. Certainly topos is not on the list but scheme, sheaf, and sheaf cohomology are. I'm sure there are a few more essential ones that are not on this list but not very many.

Second, when learning these things you are not supposed to contemplate them in their pure abstract brilliance - you're supposed to learn a whole bunch of different examples and think about what the fancy words you're saying mean in each example. If you want to study rational points on varieties I hope you already know many examples of varieties that you want to study points on - that's a good start.

Third, there is a lot of virtue here in learning things only as you need them - as long as the second or third time you need them you go back and make sure you understand them well. For instance a very large number of the important examples of schemes are varieties. One uses the language of schemes only as a new way of talking about varieties that gives you some new tools to talk about them. Again presumably you already have a reasonably good understanding of varieties so this is not some huge leap. You will not be able to get away with this forever- at some point more general classes of schemes are needed. Schemes over Z are probably the first that show up in arithmetic geometry but I'm sure the other phenomena make an appearance. However, when you encounter these concepts, you will already understand something about the notion of scheme, and it again will not be so big a leap.

Finally, let me point out that category theory is the language of algebraic geometry for a reason. When you are thinking about certain geometric ideas and trying to express them in a nontraditional setting (e.g. an arithmetic one) you will naturally be drawn to the category-theoretic concepts. This is how the language arose in the first place (although the fact that Grothendieck was around to dream up a brilliant pure abstract theory didn't hurt). To me it makes much more sense to learn category theory by first learning algebraic geometry than to do it in the other order.