I want to learn contemporary mathematical physics, so that, for example, I can read Witten's latest paper without checking other sources again and again to find some basic definitions and theorems. I know it need a long time and intensive efforts, but are there any good books related so that I can follow them in one or two years? I have learned physics theories that come before the quantum field theory, including general relativity. And I know differential geometry, category etc....

6$\begingroup$ You might want to ask at theoreticalphysics.stackexchange.com instead. $\endgroup$ – Qiaochu Yuan Dec 14 '11 at 23:13
For topics related to quantum mechanics, I recommend "Mathematical concepts of quantum mechanics" by Gustafson and Sigal.
I think the 'best' book in your scenario is "Geometry, Topology, and Physics" by Nakahara. I put this in quotes because I don't think it teaches the physicists true rigorous math (it just skims the surface with a bunch of terminology and only hands you the tools needed to compute some things in physics)... I've tried conversing with all my theoretical physics friends after they learn Algebraic Topology and Differentiable Manifolds through this text, and it's torturous. But this is just an opinon. Anyway, the textbook talks about the math that is linked to the physics, so it could benefit you to hear about the physics in this text (there is a lot!). It's considered ''mandatory'' to be on every theoretical physicist's shelf.
Other than that text, take out any textbook on String Theory (such as the classic one by Green, Schwarz, Witten). There is also the text Quantum Field Theory for Mathematicians, but I don't know how much it will be able to relate to what Witten is doing... I don't think any textbook can help you with what Witten is doing haha. Only main source seems to be the referenced papers.

7$\begingroup$ Witten has two coauthors, Green and Schwarz. The textbook by Nakahara is not considered "mandatory" by most theoretical physicists. $\endgroup$ – Jeff Harvey Dec 15 '11 at 13:48

4$\begingroup$ In my opinion, Nakahara's textbook is simply horrible in term of mathematical rigour, so it depends whether or not the OP comes from mathematics or physics. I do agree with you that it's quite torturous conversing with a physicist about geometry or topology whose only background is Nakahara's book! $\endgroup$ – Malik Younsi Dec 15 '11 at 14:06

$\begingroup$ @Harvey Sorry I should have included his coauthors for completeness. $\endgroup$ – Chris Gerig Dec 15 '11 at 18:32
Arnold's "Mathematical Methods of Classical Mechanics" Mackey's "Mathematical Foundations of Quantum Mechanics" Barut's "Electrodynamics of Fields and Particles" Stewart's "Advanced General Relativity"
For quantum field theory/particle/standard model, read the original papers and a couple undergraduate textbooks.
Remember that mathematical physics is still a branch of physics, so that if you think that everything can be reduced to mathematical axioms instead of scientific principles, then prepare to endure terrible headaches when you read works of physicists of Witten's caliber.
You can get the overview with Zeidler's multivolume QuantumField Theory I, II, III.
"Equations of Mathematical Physics" by Vladimirov V.S. is great in terms of mathematical rigour. In fact, it emphasizes on maths, not physics, and that gives better understanding of further physical problems. You can find it here