You need to read this, by Michael Atiyah:

http://www.mathunion.org/ICM/ICM1990.1/Main/icm1990.1.0031.0036.ocr.pdf

Edward Witten certainly is the master of finding impressive applications of QFT to Mathematics (for example TQFT has been instrumental for finding invariants in three-dimensional manifolds). His work has inspired many mathematicians and has created even new areas in mathematics (Seiberg-Witten theory, for example, another impressive application of QFT to mathematics). You see, luckily there are mathematicians more open minded than yourself, who are able to appreciate the invaluable insight that physics can give into very hard mathematical problems. These mathematicians will always have advantage over their more closed minded colleagues. For example, Donaldson theory is inspired in physics and in Atiyah's vision of the importance of studying the moduli space of Yang-Mills equations. Without the insight from physics it would have been very difficult to develop such theory. I recommend you to read the preface of the classical book of "Instantons and Four-Manifolds" by Freed and Uhlenbeck, where they explicitly say: "we mathematicians need physics!" and explain why.