Books about history of recent mathematics I draw on this question to ask something that has always been a pet peeve of mine. It is very easy to find books about the history of mathematics, much less so if one wants books about the recent (say > 1850) one.
Of course I know that this is difficult because not so many people would understand what's going on; to learn about the history of a subject, one should better know the subject beforehand. On the other hand, my feeling is that more or less all mathematics I know has been developed after 1850, and the growth, like in many other sciences, has been exponential. So the amount of mathematics which appears in history book seems negligible to me.
Can you point me to any good resources about the recent history of maths?
 A: I didn't see this mentioned elsewhere: the AMS has a whole series devoted to the history of mathematics, much of it fairly recent.
A: I found the book "The Genesis of the Abstract Group Concept", by Hans Wussing (I have the Dover reprint) very interesting.
It covers the development of group theory from its precursors in pre-19th century mathematics, and then traces the development of the concept of an abstract group through to the end of the 19th century.  It makes (what seems to me to be) a detailed study of primary sources, and the fact that the author has a good command of the mathematics helps lend credence to his assessment of the various historical trends and developments.  
A: The degree to which you like this answer is very dependent on the degree to which you find physics even in its own right of interest to mathematicians, but Abraham Pais is a wonderful historian of (relatively) recent physics; I'm most familiar with his biography of Einstein (Subtle is the Lord) and his history of particle physics (Inward Bound), but I understand he has many more books as well.
A: It is rare to have a history of the modern field:
Bruce Chandler and Wilhelm Magnus. The history of combinatorial group theory. A case study in the history of ideas. Studies in the History of Mathematics and Physical Sciences, vol. 9. Springer, New York, 1982.
A: For Banach space theory and linear operators, Albrecht Pietch's book History of Banach spaces and linear operators is very interesting and well-researched, and only a couple of years old.
There is also Jean Dieudonne's History of Functional Analysis.
A: I thought this book was terrific.
Labyrinth of Thought: A History of Set Theory and Its Role in Modern Mathematics: Jose Ferreiros Dominguez
I'm finding this useful in my research on a philosophy of mathematics dissertation:
Tool and object: a history and philosophy of category theory: Ralf Krömer
Others have mentioned Dieudonne's "A History of Algebraic and Differential Topology".  As a philosopher (albeit one with some graduate math training) I found it very difficult to follow.  Maybe I need to really commit to it, however.
A: Bit of a different direction: autobiographies or tributes written by mathematicians that favor mathematical events over personal ones, at least to some degree. So: 
Saunders Mac Lane
A Mathematical Autobiography
http://www.akpeters.com/product.asp?ProdCode=1500
Kadison's article on Kaplansky in:
http://www.ams.org/notices/200802/index.html
A: "A Panorama of Pure Mathematics", written by Jean Dieudonne, a book depicts a general picture of various branches of pure mathematics.
A: The Nature and Growth of Modern Mathematics written in 1970 by Edna E. Kramer is a good book that touches on a wide selection of topics - very similar to Kolmogorov, Aleksandrov, and Lavrent'ev's Mathematics: its Content, Methods, and Meaning.
A: My favorite parts of Sylvia Nasar's A Beautiful Mind (the book, not the movie) were the parts which described the Princeton math department in the 40s and 50s.
A: History, biography, and memoir are quite different genres for mathematics.   But as long as some of the latter are being recommended, I'd have to add G.H. Hardy's short memoir A Mathematician's Apology.    Like most memoirs this falls short of giving a full picture of Hardy's life and work.   His life is by now impossible to document anyway, though the somewhat fictionalized account of his years in Cambridge during Ramanujan's visit given recently by David Leavitt in The Indian Clerk is quite readable.  Leavitt's mathematics is weak at times, though corrections were made in the softcover reprint.  But he has explored the documentary evidence thoroughly, including the detailed biography of Ramanujan by Kanigel
(which however has been criticized by some Indian mathematicians for its portrayal of Indian culture and religion).
A: *

*Number theory:


"Rational Number Theory in the 20th Century: From Pnt to Flt", Wladyslaw Narkiewicz, Springer, 2011


*

*Category Theory:


"Tool And Object: A History And Philosophy of Category Theory", Ralf Krömer, Springer, 2007
and
"From a Geometrical Point of View: A Study of the History and Philosophy of Category Theory", Jean-Pierre Marquis, Springer, 2008


*

*Numerical Analysis:


"A history of numerical analysis from the 16th through the 19th century", Herman Heine Goldstine, Springer, 1977
followed by
"Numerical Analysis: Historical Developments in the 20th Century", Claude Brezinski, Luc Wuytack, Gulf Professional Publishing, 2001


*

*Operations Research


"An Annotated Timeline Of Operations Research: An Informal History", Saul I. Gass, Arjang Assad, Springer, 2005


*

*General


"Mathematical Events Of The Twentieth Century", A. A. Bolibrukh, I͡Uriĭ Sergeevich Osipov, Springer, 2006
and
"Development of Mathematics 1950-2000", Jean-Paul Pier, Birkhäuser, 2000
A: The correspondence between Cartan and Weil edited by Michele Audin contains a lot of interesting history (Bourbaki, Riemann hypothesis for curves, algebraic topology and various political topics related to mathematics).
A: I would recommend
"The Mathematician Sophus Lie: It was the Audacity of My Thinking" 
by Arild Stubhaug and R. Daly 
(original in Norwegian, there is also a German translation).
A: Two (very recent) books on (at least selected aspects of) the history of complex (holomorphic) dynamics, in the years >1850:
Audin, Michèle: Fatou, Julia, Montel. The great prize of mathematical sciences of 1918, and beyond. Translated from the 2009 French original by the author. Lecture Notes in Mathematics, 2014. History of Mathematics Subseries. Springer, Heidelberg, 2011. viii+332 pp. ISBN: 978-3-642-17853-5 (the French original is from 2009)
Alexander, Daniel S.; Iavernaro, Felice; Rosa, Alessandro:
Early days in complex dynamics.
A history of complex dynamics in one variable during 1906–1942. History of Mathematics, 38. American Mathematical Society, Providence, RI; London Mathematical Society, London, 2012. xviii+454 pp. ISBN: 978-0-8218-4464-9 
A: Nicolas Bourbaki's Elements of the History of Mathematics covers the history of mathematics, and is a nearly unaltered reproduction of the historical notes from Bourbaki's Elements of Mathematics.  The historical notes themselves are a great source for this type of information, and this book collects them in a nice readable format.  It is pretty funny the way Bourbaki writes about the contributions of members of the group (from the chapter on uniform spaces: "Uniform spaces have only been defined in a general way recently by A. Weil...".  He also plays up the importance of filters, in classic Bourbaki style.). 
Edit: Fixed.
A: I enjoyed the book "Remarkable Mathematicians: From Euler to von Neumann" by Ioan James.  It gives a good account of what the mathematicians were doing (in their personal lives and professional) and how their interactions shaped mathematics.  It is fairly light on the mathematical content but an enjoyable read nonetheless. 
A: The Shape of Inner Space: String Theory and the Geometry of the Universe's Hidden Dimensions
(2010)
by Shing-Tung Yau and science writer Steve Nadis.
It is an interesting mixture of autobiography, history of a slice of mathematics (largely surrounding 
Calabi-Yau manifolds), and popular-science tutorial.  Here is Witten's endorsement:

"Shing-Tung Yau and Steve Nadis take the reader on a fascinating tour of many contemporary topics in geometry and physics. Readers will find many challenging ideas to explore in this book, and even specialists will enjoy Yau’s reminiscences about his education and work."

A: The ICM website records all the addresses given at the Fields Medal conferences, and they always begin with a short paper by a distinguished mathematician describing to a "broad" audience why that medallist has deserved the award. This is often very interesting, containing recent historical remarks. 
You can read Katz' description of Deligne's work in 1978, for instance: I find it delightful. Funny enough, the very same conference saw a contribution by André Weil
http://www.mathunion.org/ICM/ICM1978.1/Main/icm1978.1.0227.0236.ocr.pdf
entitled "History of Mathematics: Why an How" which addresses many of your questions – and answers them, I think. For example, he discusses "why" is a mathematician interested in the history of "which" mathematics, and "how" this should be pursued. And the very last paragraph begins: "What, then, separates the historian from the mathematician when both are studying the work of the past?" Besides all, extremely enjoyable!
A: This small book is quite interesting. It gets you a fairly wide historical perspective and also key highlights in math done by Erdos and his colleagues.
http://www.amazon.com/MY-BRAIN-OPEN-Mathematical-Journeys/dp/0684859807/ref=sr_1_2?ie=UTF8&s=books&qid=1273174664&sr=1-2
A: Michèle Audin's book on the history of complex iteration has already been mentioned. But "Remembering Sofya Kovalevskaya" should be mentioned also.
A: 
Jean-Pierre Marquis and Gonzalo E. Reyes The History of Categorical Logic 1963–1977 

The above article discusses the history of the development of categorical logic starting from a brief historical sketch of the birth of Category Theory and its early developments.  
A: Here are a few books on the history of recent mathematics that I recommend:
A History of Algebraic and Differential Topology, 1900 - 1960 by Jean Dieudonne.
History of Topology by I.M. James.
Reciprocity Laws: From Euler to Eisenstein by Franz Lemmermeyer. 
The Shaping of Arithmetic after C.F. Gauss's Disquisitiones Arithmeticae
by Catherine Goldstein, Norbert Schappacher, and Joachim Schwermer.
The Mathematical Coloring Book: Mathematics of Coloring and the Colorful Life of its Creators 
by Alexander Soifer.
A: Multiple answers have mentioned books by Dieudonne. I can recommend one more: Mathematics- The Musics of Reason (Springer, 1992) or the French original Pour l'honneur de l'esprit humain: les mathématiques aujourd'huis (Hachette, 1987). Its stated purpose is to give a generally accessible account of major mathematical developments after 1800, of which it does a remarkably good job.
A: Jeremy Gray, Plato's Ghost: the modernist transformation of mathematics, Princeton 2008 discusses the changes in mathematical thinking and views of mathematics between 1890 and about 1930.
Dennis E. Hesseling, Gnomes in the Fog: The Reception of Brouwer’s Intuitionism in the 1920s, Birkhauser 2012 (reprint of the 2003 edition). I haven't read this one, but it deserves a look on the basis of the title alone.
(Forthcoming in Nov 2017) Alberto Cogliati, Writing Small Omegas: Elie Cartan's Contributions to the Theory of Continuous Groups 1894-1926 (Studies in the History of Mathematical Inquiry) looks as though it could be very interesting.
Piergiorgio Odifreddi The Mathematical Century: The 30 greatest problems of the last 100 years translated by Arturo Sangalli, forward by Freeman Dyson, Princeton University Press 2004 is very readable.
A: Jean Dieudonne's history of Algebraic Geometry is fantastic.  It has been helping me put a lot of things in perspective.
A: A very enjoyable read of modern topic is Weibel's A History of Homological Algebra (40 pages).
A: Another book which covers some recent history in its last chapters is The Queen of Mathematics: A Historically Motivated Guide to Number Theory. Also look at 
A History of Abstract Algebra by Israel Kleiner
Episodes in the History of Modern Algebra (1800-1950)
You do not have to restrict yourself to books, there also articles that touch on the history of a particular area of mathematics.
A: Other examples (just to give an idea of the choice), thematic sample,
with scholarly work, popular science, and other types :
Recent theorems
Four Colours Suffice: How the Map Problem Was Solved de Robin J. Wilson
Proofs and Confirmations: The Story of the Alternating Sign Matrix Conjecture de David M. Bressoud, William Watkins, Gerald L. Alexanderson, and Dipa Choudhury
Kepler's Conjecture: How Some of the Greatest Minds in History Helped Solve One of the Oldest Math Problems in the World de George G. Szpiro
20th century
The Honors Class: Hilbert's Problems and Their Solvers de Benjamin H. Yandell
Mathematical Analysis During the 20th Century de Jean-Paul Pier
Development of Mathematics 1900-1950 de Jean-Paul Pier
The Mathematical Century: The 30 Greatest Problems of the Last 100 Years de Piergiorgio Odifreddi
Biography
Ludwig Wittgenstein: The Duty of Genius de Ray Monk
Von Neumann, Morgenstern, and the Creation of Game Theory: From Chess to Social Science, 1900-1960 de Robert J. Leonard
The Random Walks of George Polya de Gerald L. Alexanderson
Logic's Lost Genius: The Life of Gerhard Gentzen de Eckart Menzler-Trott
Auto biography
Indiscrete Thoughts de Gian-Carlo Rota et Fabrizio Palombi
Discrete Thoughts: Essays on Mathematics, Science, and Philosophy de M. Kac
A Mathematician Grappling With His Century de Laurent Schwartz
= "Un mathematicien aux prises avec le siecle" original french title
Photographs
Mathematical People: Profiles and Interviews de Donald J. Albers
Source books
(because it is invaluable to read the original articles)
From Frege to Godel: A Source Book in Mathematical Logic, 1879-1931 de Jean van Heijenoort
Yearly compilation
What's Happening in the Mathematical Sciences 200x-200x+1 de Barry Cipra
Math and society
The Rise of Statistical Thinking 1820-1900 de Theodore M. Porter
other
Abrégé d'histoire des mathématiques, 1700-1900
6000 Jahre Mathematik: Eine Kulturgeschichtliche Zeitreise - 2. Von Euler Bis Zur Gegenwart de Hans Wussing
A: I don't think that there is a shortage of books on "recent" history of mathematics - if anything, the growth has been exponential here as well! There are many recent books dealing with more specialized areas written by eminent scholars, e.g. 
Charles Curtis, Pioneers of representation theory: Frobenius, Burnside, Schur, and Brauer
Armand Borel, Essays in the history of Lie groups and algebraic groups
Thomas Hawkins, Emergence of the theory of Lie groups. An essay in the history of mathematics 1869–1926
and, although this is not a book on history of mathematics as such, the erudite
Marcel Berger, A panoramic view of Riemannian geometry
Among broader views, I enjoyed reading 
Piergiorgio Oddifreddi, The mathematical century. 30 greatest problems of the last 100 years
The four-color problem, Kepler's conjecture, and the Monster have all been featured in popular mathematics books.
A: These have already been mentioned in comments, but I wanted to put them here explicitly:
I Want To Be a Mathematician: An Automathography by Paul Halmos
I Have a Photographic Memory by Paul Halmos
And as for photographs, the recent Mathematicians: An Outer View of the Inner World is another good one.
There's some good historical material in the Princeton Companion to Mathematics as well.
A: Monastyrsky's Modern Mathematics in the Light of the Fields Medals. For a review see here.
A: Andre Weil wrote an autobiography, The Apprenticeship of a Mathematician.
A: Leo Corry - Modern Algebra and the Rise of Mathematical Structures (here is a review)
A: Surprised the Grothendieck-Serre letters haven't been mentioned yet.  As historical primary sources go, they don't get much better than that.
A: Mark Ronan's book, Symmetry and the Monster, deserves a mention. It is a very well-written, popular account of the classification of finite simple groups which even the experts can learn from.
A: I am shocked that Carol Parikh's "The Unreal Life of Oscar Zariski" hasn't been mentioned.
It paints a very charming picture of the man and casts new light onto luminaries like Weil, Lefschetz and Birkhoff.
A: For a well-written historical presentation of the mathematical physics underlying basic quantum mechanics
The Rise of the New Physics by A. D'Abro
A detailed, fascinating presentation on some aspects of the evolution of mathematical physics is the book
Masters of Theory: Cambridge and the Rise of Mathematical Physics by A. Warwick
