When I was 27 I was a lugger in a meat market. I got my PhD at 35, and spent the next 33 years on the faculty at UGA. To be honest, getting a PhD was hard and not especially lucrative, and certainly not everyone in my graduate class has earned a living doing math at the college level. But I would not have wanted to do anything else. I have met many brilliant and generous people and been privileged to discuss math with them all these years. If you pursue your goal but don't find a suitable college teaching job, and you like discussing math with interested kids, I suggest looking at private high schools. Some of my most rewarding teaching occurred when I volunteered one year at a good local high school.