As an actuary, if you have passed a reasonable number of exams, you will have a proven record of mathematical ability. Most graduate programs should be familiar with actuarial science, and it might help if you apply to a school that does have an actuarial program (housed in math preferably), even if that's not what you're planning to specialize in.

There shouldn't be any resistance from the math department; there might be some administrative issues from the university-wide graduate school, which is usually the place where admissions take place. But in general, if the Math dept is interested, they'll be able to help you navigate these complications.