In my late 20's I was a lugger in a meat market.  I got my PhD in my mid 30's, and spent the next 3 decades on the faculty at UGA.  To be honest, getting a PhD was hard and not especially lucrative, and certainly not everyone in my graduate class has earned a living doing math at the college level.  But I would not have wanted to do anything else.  I have met many brilliant and generous people and been privileged to discuss math with them all these years.  If you pursue your goal but don't find a suitable college teaching job, and you like discussing math with interested kids, I suggest looking at private high schools.  Some of my most rewarding teaching occurred when I volunteered one year at a good local high school.