You need to read this, by Michael Atiyah:
http://www.mathunion.org/ICM/ICM1990.1/Main/icm1990.1.0031.0036.ocr.pdf
Edward Witten certainly is the master of finding impressive applications of QFT to Mathematics (for example TQFT has been instrumental for finding invariants in three-dimensional manifolds). His work has inspired many mathematicians and has created even new areas in mathematics (Seiberg-Witten theory, for example, another impressive application of QFT to mathematics). You see, luckily there are open minded mathematicians who are able to appreciate the invaluable insight that physics can give into very hard mathematical problems. These mathematicians will always have, in my opinion, advantage over the rest of their colleagues. For example, Donaldson theory is inspired in physics and in Atiyah's vision of the importance of studying the moduli space of Yang-Mills equations. Without the insight from physics it would have been very difficult to develop such theory. I recommend you to read the preface of the classical book of "Instantons and Four-Manifolds" by Freed and Uhlenbeck, where they explicitly say: "we mathematicians need physics!" and explain why. Of course, I am not implying that physics is useful for every mathematical problem, but it is indeed so for a good number of problems in geometry.