No, it's certainly not too late. The success is not guaranteed to anyone in mathematics, but from what you describe it looks like your chances to become a mathematician are not fewer than anyone else's. I would encourage you to devote your time to the study of what you find most interesting, having also in mind that a broad education in mathematics is very useful for a (prospective) mathematician.