I did a Mathematics Batchelors 7 years ago, but was always having difficulty trying to cram everything in or understanding things. Naturally, this didn't pan out too well. But looking back on the notes now, that daunting information seems a lot more readable than it did before exams.
So, in summary, no, I don't think you're too old. It would appear that the myths of being too rusty are bunk. Besides, when you were a lot younger, you might have been interested in just partying to understand anything properly! quite a lot of undergrads I knew just memorized proofs and exercise questions to get by, but didn't quite get an intuitive feel for it.