I started at 32, almost 33. I've been worried that I've lost my best years but I'm getting through Analysis fairly well, teaching myself topology and preparing to take number theory & a real probability course in the spring.

In my second semester the topology (Munkres) and the Analysis (Rudin) were beyond me, but just a bit of work went a long way. Much of Concrete Math (Knuth) was beyond me last year as well, but I'm able to get what I want from it as I need to.

I'm slower in a lot of ways than I was when I was 18, but as some have said maturity does a lot for a person. There's no way I'd have done as well as I am then. Also, speed isn't what's important for mathematics. It's great if you have to take the Physics GRE and need to calculate on the fly, but I've found I tend to be able to grok proofs at a deeper level than I would have.

Also, professors love the work ethic us "older" people have.

The only suggestion I can offer is find a program that you fit well in, whether big or small, with strong guidance or with a lot of independent study and find a professor you enjoy working with if at all possible.

I couldn't be happier and my life couldn't be better were I to have chosen a different path. Give it a try, the worst that happens is you decide it's not for you.