Research mathematics is a small world. It is rather likely that the people who are currently editing and refereeing your paper will play a significant role in the development of your career in the next 5 years or even more. They might not take too kindly to the withdrawal, especially if they think that the move was not really warranted (i.e., if they don't think that your result is really much too good for the journal you submitted). So think to yourself: is this a risk worth taking? Is your result that badly served by the current journal?

As for the impact factor, have you really tried to find out what people think about those? Deans and other paper pushers might put a lot of store by them, and so it's definitely something to keep in mind in the long run, but your colleagues' opinion of a journal is not based on a somewhat artificial numerical construct, so for your immediate future, the actual name and reputation of the journal is much more important than a somewhat arbitrary numerical score. (It is a fact that the math journals with the highest impact factor are not necessarily the most prestigious ones.)