I am a theoretical physics major student working on string theory. I want to understand the work of MF Atiyah and R Bott, "The Yang-Mills equations over riemann surfaces" . What kinds of mathematical background does it need? books or papers? (I only learned Nakahara's book on geometry) Thanks in advance.
This is one very beautiful, influential and very challenging paper. You need to know differential geometry, some topology (cohomology, characteristic classes a bit of Morse theory), some algebraic geometry, and a bit of analysis (elliptic operators and complexes).
There is no one book that contains all of these, though Frankel's book The Geometry of Physics would be a good book to have by your side. Perhaps the best way to go is start reading the paper and whenever you get stuck get help from books and, better yet, an experienced mathematician. Early on, you'll get stuck quite often, so you need to stay cool and persevere. It beats spending a year reading all the required material. In any case, I want to emphasize that having adult supervision would make this experience more bearable and much more enriching.
Another good place to start is Donaldson's 8 page paper "A new proof of a theorem of Narasimhan and Seshadri" (available here) - this is a little less bulky than Atiyah-Bott but has the same flavour and concerns the same circle of ideas. Following Liviu Nicolaescu's advice (reading this paper and looking up what you need when you need it, e.g. stability of holomorphic vector bundles) will get you a long way with this paper and you could finish it in finite time (whereas Atiyah-Bott would take a lot longer). Don't be fooled, though - 8 pages of Donaldson doesn't equal 8 pages of Harry Potter. It could take a month or more.
If it helps (and it might not) I gave a graduate-level course last year which covered some of the underlying ideas, explained the Donaldson result and made some headway into the Atiyah-Bott paper. You can read the notes and see the lectures online: http://www.homepages.ucl.ac.uk/~ucahjde/yangmills.htm These may not be terribly useful depending on your background. At least I collated some of the basic stuff about holomorphic vector bundles and stability and a proof of the all-motivating Kempf-Ness theorem (in its simplest form).
Edit: I think a problem with reading either of these papers is that they're written in very condensed notation. It helped me to rewrite things for myself with a bit more notation (dare I say indices) to understand what each line means.