There are two separate questions you might be asking here:
1) How much should one care about having bad ratings relative to average ratings
2) How much should one care about having average ratings relative to good ratings
I think in the first case you should care a lot more. If your main goal is to get a job at a research institution then my guess is that there are more doors that close from being a bad teacher than there are doors that open by being a great teacher.
Furthermore, not all aspects of teaching are reflected in ratings and you could certainly be a good teacher and get ratings that aren't great. I'm mostly speaking from the experience of evaluating mentors at Mathcamp, where I have the most experience comparing student ratings to the independent evaluation of teaching quality by observers. There are lots of other factors that influence ratings beyond teacher quality, and in the middle of the curve these other factors make a bigger difference than quality. That said, both at Mathcamp and at Berkeley, it seemed the great teachers got great evaluations, and terrible teachers got terrible evaluations. At Mathcamp when we had teachers who we thought were good who only got mediocre evaluations we were generally willing to overrule the students opinions, and that's in a situation where our primary concern is teaching.
