You need to read this, by Michael Atiyah:

http://www.mathunion.org/ICM/ICM1990.1/Main/icm1990.1.0031.0036.ocr.pdf

Edward Witten certainly is the master of finding impressive applications of QFT to Mathematics (for example TQFT has been instrumental for finding invariants in three-dimensional manifolds). His work has inspired many mathematicians and has created even new areas in mathematics (Seiberg-Witten theory, for example, another impressive application of QFT to mathematics). You see, luckily there are mathematicians who are able to appreciate the invaluable insight that physics can give into very hard mathematical problems. These mathematicians may have, in my opinion, advantage over the rest of their colleagues. For example, Donaldson theory is inspired in physics and in Atiyah's vision of the importance of studying the moduli space of Yang-Mills equations. Without the insight from physics it would have been very difficult to develop such theory. I recommend you to read the preface of the classical book of "Instantons and Four-Manifolds" by Freed and Uhlenbeck, where they explicitly say: "we mathematicians need physics!" and explain why. Of course, I am not implying that physics is useful for every mathematical problem, but it is indeed so for a good number of problems in geometry.

Note added: let me give a very explicit example of how QFT (in this case through String Theory) strongly influenced mathematics (in this case algebraic geometry) and indeed solved an outstanding open problem in AG. I am of course talking about mirror symmetry. In the following paper:

https://www.staff.science.uu.nl/~beuke106/HypergeometricFunctions/COGP.pdf

Philip Candelas, Xenia C. DE LA Ossa, Paul S. Green and Linda Parkes computed and predicted, using a mix of String Theory, Supergravity and SCFT, the number of rational curves on a generic Quintic three-fold. This is the basis of Mirror Symmetry. Algebraic geometers were flabbergasted. The paper is indeed not easy to understand neither for mathematicians nor for string theorists (for general physicists is simply out of any reach), since it mixes the previous mentioned areas with relatively sophisticated mathematical arguments. Algebraic geometers immediately jumped into the topic, trying to understand what the heck was going on. Without String Theory, QFT and Supergravity, the outstanding conjecture encoded in Mirror Symmetry most probably would have not been proposed in "our era" (as I have heard from various algebraic geometers).