This is more of a comment than an answer, but I think it bears saying, based on some of the comments at the top.  Anyone who is now a math grad student was not a typical member of an undergrad math class.  While clearly the correlation between evaluations and good teaching is obviously not perfect, it is not negligible either.  In my experience (which includes many years of recieving evaluations, at several different institutions, many years of reading teaching letters, and other related considerations of this question), if someone gets very low evaluations, there is typically a good reason for it.  While it may not be easy to get top evaluations, reasonable teaching almost always avoids terrible ones.  

And when one is recruiting post-docs, or more senior candidates, one wants to have some assurance they won't be a teaching disaster.  If they are a brilliant teacher, all the better,
but is more concerned about avoiding disasters, since these lead to trouble and difficulty for everyone.  

If you are applying to research positions, if your teaching is average or better, this should be fine.  If your evaluations are consistently and significantly below average, this is probably a sign that you need to put some effort into improving your teaching.  Students normally react well to clear explanation, sincere treatment, some humility on the part of the lecturer, and a little effort to be organized.  Negative evaluations often correlate with the violation of one or more of these principals.    Very good teaching do much more, but just doing a reasonable job should be enough to keep your evaluations reasonable.