You should worry about your teaching evaluations before you get them. In other words, you should put effort into your teaching as 1) it does matter at almost every institution, and 2) you are teaching people math, and that is important. It is also a good idea to discuss the evaluations with your students before you hand them out. Remind them that this feedback will help you to improve your teaching so you would appreciate them telling you what worked and what didn't. Often, only "angry" students will respond in the comment sections. By taking five minutes and reminding the students that the evaluations are a tool for communication, you should get much better feedback.

As for worrying about them afterward, it is true that they are a secondary consideration at most research universities. However, it is not helpful to have anything on your application that puts you lower on the list. If your evaluations are bad, make sure that the person who writes your teaching letter speaks about the positive aspects of your teaching. If you have to send in the evaluations, then have them address directly the mitigating factors (if there were any) that caused the bad evaluations.