I'm a faculty member who reviews many applications. I think that your actuarial experience would actually be a positive factor in your application to some graduate programs (particularly for programs with professional master's degrees and applied programs that focus on applications in financial mathematics, applied statistics, etc.), but it would not be helpful (and might even hurt) in other cases (e.g. the top PhD programs in pure mathematics.)
I would encourage you to either start taking graduate courses at a local university on a part time basis, or to enter a full time MS program with the intention of transferring to a PhD program after completing the MS.