I strongly recommend finding a good advisor (someone who you get along with, who has compatible understanding of how hands-on the advisor will be, who will keep you funded, who can get you a postdoc, who actually wants a student etc.) over choosing a particular subfield.  There's too little correlation between what you learn about a subject as a starting graduate student and what it's like to do research in that field for you to make decisions on that basis.  That said, it's reasonable to narrow your search to broad fields (say "I want to do algebra") because that still allows a huge range of subfields to work in.  As long as your tastes are reasonably broad (and if they're not you should work on broadening them) you and your advisor should be able to find something that interests you both for you to work on no matter what your advisor's particular speciality.