Currently I'm a PhD student in applied math at a fairly good research university. I just wish to share some of my experiences which might help you make the decision.
My undergrad education was in Electronics Engineering. That's when I truly fell in love with math. I did extremely well on math courses and extremely poorly on the rest, decided that I was best suited for math and decided to pursue grad studies in math. After my bachelor's degree, I spent about a year and a half studying undergrad math (mostly advanced calculus, analysis, ODE's, linear algebra and modern algebra). I took the GRE subject test and did average on that. Looking back, I realize now that I should have spent at least a year more making more intense preparations. But back then, I was working alone all by myself, without anybody to guide me through my difficulties, and without any clear idea about the overall picture and structure of mathematics.
I was able to get admission into MS applied math program in my current university. At the end of my 1st year as an MS student, I had passed both my PhD qualifiers and decided to go for PhD. That has turned out to be the worst decision I have ever made in my life.
The problem was that I was, and I still am, in love with pure math and I overlooked the fundamental differences between my own intellectual preferences and the research work that was going on in our department. Although our applied math dept is quite renowned, it does not contain any professor who has a PhD in mathematics. Most of them are physicists, some of them are biologists, some computer scientists and some statisticians. Now, I have all the respect in the world for these disciplines, but in the end my work turned too far away from real mathematics. Today I spend about 95% of my research time doing C/C++ coding, about 3-4% physics and the rest is math.
There was a time when I was seriously under the impression that I was going to become a professional mathematician at the end of my PhD. However, a few years down the road, I realized that I am not even close to being a mathematician. It became clear that if I want to become a professional mathematician, I'm going to have to pursue another PhD, this time in pure math without any compromises.
Today I'm carrying on with my PhD in applied math only because I have spent too much of time and effort to turn away from it. And I have full intention of going for a second PhD in pure math. I have learned some of the biggest pitfalls in this process, and I have learned them the hardest way possible.
During all this ordeal, my love for math, especially theorem-proof type math and problem solving in general, has grown tenfolds. And I realize today that I will never be happy without mathematics. I am almost 31 today, but I'm determined to go for pure math.
My experience says that you don't necessarily become slow as you grow old, but it certainly becomes difficult to commit yourself 100% to math. Especially if you have a family (as I do), and if your family members don't understand your passion for math, then things can get a little rocky at times. But that next proof, that next eureka moment, that next beautiful problem make it all worth it!