If you are viewed as a star research mathematician, I think it is safe to say that your student evaluations will do absolutely no damage to you at all.

If, however, you are viewed as anything less than a star and have to compete against many other candidates of comparable quality for a tenure track position at a school other than the top ranked schools, then the department will care a lot about whether you will be a good citizen of the department. This is only sensible, since if they eventually give you tenure, they're most likely stuck with you for life. Not provoking complaints by students to the university administration about your teaching is perhaps the most important aspect of being a good citizen.