It's true the atmosphere of a school make a big difference. Without knowing anything more, I would suggest applying to a variety of schools, then visit the schools you get into. This will give you a much better sense of whether or not the school is a good fit for you than just the ranking. I think after visiting, you should be able to narrow your choice down to a couple schools you like, and at that point, probably either would be a good choice.

Incidentally, I wrote some tips on visiting grad schools after I visited quite a few. You might also check out the career advice section on Terry Tao's blog, though I'm not sure there's anything specific about choosing a grad school.