The pace issue is a misconception of yours. You will have to learn the graduate level mathematics as quickly as is necessary to begin your dissertation work if you are going to a PhD program. Essentially, you have to catch up to a century's worth of mathematics in a year in and a half. So deal with it.
But to answer the original question, what will be essential to your survival in grad school and beyond is your relationship with your PhD advisor. I have seen students from not top programs land better jobs because their advisors took the time and care to help their students, bring them to conferences, introduce them to others working in the field, help them select post-doc positions, etc.
If you know the area in which you want to work, then you should select grad school based on that. If you are not sure, then you should select among the best to which you are admitted, and select an advisor based upon her area of expertise and her ability to get grad students into good schools. Also check out the social network among the grad students. Is it solid? Will others help you in your studies.
Ultimately, what makes your dissertation great is your own creativity and your depth of thought, but all else being equal, a good advisor is as important as the school that she teaches at.