I'm an assistant professor at the University of South Carolina. Although we are not Harvard or Princeton, we are a good research department with a solid Ph.D. program, and many of our graduates are successful both within and outside academia. Several of our graduate students are older than I am, which I think is great!

The graduate admissions committee will look for firm evidence that you are prepared to succeed in graduate school. In particular, you will need evidence that you currently have mathematical ability and motivation comparable to other entering grad students. To this end, I don't really see any way around getting recommendation letters.

I strongly second Suvrit's advice -- I was out of school for a couple of years before grad school, and I did the same thing. I would also encourage you to do the homework, take the exams (if any), and ask the professor to take at least a brief look at your work. Although the professor is not obligated to grade it, I think many would be willing to do this. If you do a good job, your professor will be able to write you a good letter (and also give you useful advice).

I might add that you will want to demonstrate knowledge of foundational topics, in particular algebra and analysis. If the courses you sit in on aren't in these topics, or don't have them as prerequisites, you might ask your letter writers to spend a half hour vetting your background knowledge so they can address it in their letters.

Good luck!