I am a theoretical physics major student working on string theory. I want to understand the work of MF Atiyah and R Bott, "The YangMills equations over riemann surfaces" . What kinds of mathematical background does it need? books or papers? (I only learned Nakahara's book on geometry) Thanks in advance.

3$\begingroup$ The standard approach is to look up key words in standard references (e.g. Wikipedia if you have to, or Google) and see what texts they recommend, or even do some reference tracing the oldfashioned way. $\endgroup$– David Roberts ♦Oct 25, 2012 at 13:34
3 Answers
This is one very beautiful, influential and very challenging paper. You need to know differential geometry, some topology (cohomology, characteristic classes a bit of Morse theory), some algebraic geometry, and a bit of analysis (elliptic operators and complexes).
There is no one book that contains all of these, though Frankel's book The Geometry of Physics would be a good book to have by your side. Perhaps the best way to go is start reading the paper and whenever you get stuck get help from books and, better yet, an experienced mathematician. Early on, you'll get stuck quite often, so you need to stay cool and persevere. It beats spending a year reading all the required material. In any case, I want to emphasize that having adult supervision would make this experience more bearable and much more enriching.

$\begingroup$ Thanks for your suggestions. They are very helpful to me not only in academy but also outside of it. $\endgroup$ Oct 25, 2012 at 14:58
Another good place to start is Donaldson's 8 page paper "A new proof of a theorem of Narasimhan and Seshadri" (available here)  this is a little less bulky than AtiyahBott but has the same flavour and concerns the same circle of ideas. Following Liviu Nicolaescu's advice (reading this paper and looking up what you need when you need it, e.g. stability of holomorphic vector bundles) will get you a long way with this paper and you could finish it in finite time (whereas AtiyahBott would take a lot longer). Don't be fooled, though  8 pages of Donaldson doesn't equal 8 pages of Harry Potter. It could take a month or more.
If it helps (and it might not) I gave a graduatelevel course last year which covered some of the underlying ideas, explained the Donaldson result and made some headway into the AtiyahBott paper. You can read the notes and see the lectures online: http://www.homepages.ucl.ac.uk/~ucahjde/yangmills.htm These may not be terribly useful depending on your background. At least I collated some of the basic stuff about holomorphic vector bundles and stability and a proof of the allmotivating KempfNess theorem (in its simplest form).
Edit: I think a problem with reading either of these papers is that they're written in very condensed notation. It helped me to rewrite things for myself with a bit more notation (dare I say indices) to understand what each line means.
Just as a reference, there was a summer course given by Alex Waldron at USTC here on "YangMills Equations over Riemann surface", and his aim is to explain the proof of Donaldson's paper from the most basic differential geometry.