I would not necessarily say that you're too old to learn mathematics. I'd even say it may be an advantage. Many mathematical problems you'll encounter during during your studies can be solved easily if you accept and understand how mathematicians proceed when they come across problems. This normally requires one thing: patience. It is normal to be stuck on a problem for days or even for weeks (my experience sometimes!). However, many (young) people tend to becoming very frustrated if a solution isn't obvious. Mostly, indeed, it isn't. These are then the interesting problems. There is a saying that the older you are the more patient you are. If you combine this with motivation and passion about mathematics and still some curiosity, you'll have no problem in succeeding (no guarantee, though, but it becomes more likely ;) ).
My opinion is that you never become too old to do something academic. My dad for instance wanted too learn something about electrical engineering. So, he read some books about it and is able to solve the problems these books contain. And, he studied law and is 55 years old. I think this shows that with enought motivation you can do anything. Perhaps you'll have a decreased memory capacity compared to your fellow students, but only perhaps. I think, you never know, if you don't try. So just try it.