Go for it! In any case, if you can't help but study math all the time, you may become a good mathematician whether you want to or not :-)

If you need more encouragement, read the supportive answers to the MathOverflow question "Too old for advanced mathematics?".