If you are a woman or a member of an "under-represented group" in mathematics, then there is some evidence to suggest that your teaching evaluations may be lower than they should be. The literature is equivocal and you can do your own research by googling around on "implicit bias" and "student evaluation of teaching." One principle seems to be that if you don't fit the implicit picture of a mathematician -- that is, you're not a geeky white male -- then implicit bias causes students to suspect your competence and rate you lower.
I've heard a variety of ways of counteracting this -- I've heard people urge women and members of other under-represented groups to dress more professionally in class then your typical white male professor does, so they are seen as more competent. For graduate students, having the professor who's teaching the class overall formally endorse your competence by introducing you at the beginning of the semester (this is so-and-so, a highly talented graduate student in mathematics with an undergraduate degree for XXX; she'll be running your discussion section) is probably a good idea.
My own view is that if you are teaching and you sense from the atmosphere in the class that you're not getting the respect you are entitled to by virtue of your academic achievements, you should consult immediately with someone in power. Don't let it get away from you.