Follow your dreams!

One thing you should be aware of is that there is a disconnect between the math that non-mathematicians are familiar with --- everything in calculus and below --- and the stuff that comes after. A lot of people are surprised when they get there.

The stuff that comes after calculus is very abstract. There's no simple procedure to get the answer. Every problem is different, and you are expected to be clever and figure out how it's done. This does not adequately express what it's like. Some people find it beautiful and hypnotic, and it's easily enough to keep a person occupied for the rest of their life. But for other people it is terrifying.

If you are really "in the groove of it," then doing higher math is an extremely gratifying experience, unlike anything else in life. But a lot of people find that math is more than what they bargained for, and if you struggled in your college math classes, this might be something to think about. I do not wish to discourage you; just to convey some idea of what you are signing up for. Whatever you decide, I wish you great success!