Fiction books about mathematicians? What are some fiction books about mathematicians? 
It seems to me rather difficult for writers to create good books on this subject.
Some years ago I thought there were no such books at all.
There are many reasons: it is difficult to describe the 
process of discovery and describe it in the exciting way. 
The subject has narrow audience and not the way to make best-seller...
Comments on how authors try to avoid these problems are also welcome.
The movie "A Beautiful Mind" is a (beautiful for me) example, where the story of mathematician was mixed with love and spy stories to make it interesting for general
audience, well not so much  preserved from mathematician's story, but nevertheless
I am quite positive about it.
Here is a related MO question:
Movies about mathematics mathematicians
 A: Rebecca Goldstein's The Mind-Body Problem is set in the Princeton math department, with some of the characters pretty clearly based on their real-world counterparts.
A: You must take a look at the following thoroughly list for Mathematical fiction http://kasmana.people.cofc.edu/MATHFICT/all.php
Finally, let me just mention a recent novel about Hardy, Ramanujan, Littlewood, etc. The title is "The Indian Clerk: A Novel" and the author is "David Leavitt" .http://www.amazon.com/The-Indian-Clerk-A-Novel/dp/1596910402
A: I think that many, if not all, short stories of Jorge Luis Borges qualify. Even if they're not directly about maths, they often involve some kind of strange "mathematical structure", like paradoxes, symmetries, mirrors, labyrinth, distortions of space and time. Also, the notion of infinity is a common topic of these stories: the most well known example is "The library of Babel", but there are many other examples like "The immortal" or "Aleph" (which is of course a reference to the standard notation for transfinite numbers).
A: A fictionalized L. Kantorovich is a leading character in Red Plenty, which the author characterizes as "not exactly history and not exactly a novel, but a fusion of the two".
A lengthy, semi-technical review (which won a web-award for science writing) can be found at Crooked-Timber.
A: I recommend The Number Devil, an amazing book for small children about a boy bored with mathematics who is approached in his dreams by a little red man, the number devil, who teaches him mathematics. Although a children's book, I think this certainly counts.
A: Le Théorème du Perroquet, by Denis Guedj (1998) has been successful in France.
Translated as The Parrot's Theorem

A: Professor Moriarty - in Sherlock Holmes compendium.

He is a man of good birth and excellent education, endowed by nature with a phenomenal mathematical faculty. At the age of twenty-one he wrote a treatise upon the binomial theorem which has had a European vogue. On the strength of it, he won the mathematical chair at one of our smaller universities, and had, to all appearances, a most brilliant career before him. But the man had hereditary tendencies of the most diabolical kind. A criminal strain ran in his blood, which, instead of being modified, was increased and rendered infinitely more dangerous by his extraordinary mental powers. Dark rumours gathered round him in the University town, and eventually he was compelled to resign his chair and come down to London. He is the Napoleon of crime, Watson. He is the organiser of half that is evil and of nearly all that is undetected in this great city...
—Holmes, "The Final Problem"

MacTutor adds:

He is the celebrated author of "The Dynamics of an Asteroid", a book
which ascends to such rarefied heights
of pure mathematics that it is said
that there was no man in the
scientific press capable of
criticizing it.
He was also an early exponent of the
subject of Game Theory, well in
advance of Nash and Von Neumann. Oskar
Morgenstern analysed his contributions
in [3].

A: Another play, "Arcadia" by Tom Stoppard. One of the main characters is Thomasina Coverly,  "a precocious teenager with ideas about mathematics well ahead of her time" (to quote from Wikipedia). It is not too much of a spoiler to reveal that this fictional young woman came up with what we now call fractals.
A: "The Housekeeper and the Professor" gives an interesting insight into the though processes of a
Mathematician, written from the perspective of a non-Mathematician. The author sem to me to have a good understanding of the former.
A: I recommend books by the great Polish hard s-f writer Stanisław Lem.
"His Master's Voice" is a novel about scientists who receive the message from extraterrestrial civilization and try to decipher it. The narrator of the story is a mathematician named Peter Hogarth.
Another great book by Lem in which you can find a lot of reference to mathematics is "The Cyberiad", a series of short, humorous stories about adventures of two constructors, Trurl and Klapaucius. "The dragons of probability" is a masterpiece!
You can find out more about Lem's books here: https://english.lem.pl/
A: One more: Maths à Mort, by Margot Bruyères (Wayback Machine). This one is a thriller that takes place in the IHES.
A: Definitely Maybe, where the mathematician is modeled after a famous Russian mathematician.
His Master's Voice, where the main character is a mathematician.
A: The First Circle by Solzhenitsyn features a mathematician as the main character.
A: Measuring The World [Large Print] [Hardcover]
Daniel Kehlmann
Daniel Kehlmann (Author)
Carol Brown Janeway (Translator)
From the book description on Amazon's page:
Measuring the World recreates the parallel but contrasting lives of two geniuses of the German Enlightenment - the naturalist and explorer Alexander von Humboldt and the mathematician and physicist Carl Friedrich Gauss. Towards the end of the 18th century, these two brilliant young Germans set out to measure the world. Humboldt, a Prussian aristocrat schooled for greatness, negotiates savannah and jungle, travels down the Orinoco, climbs the highest mountain known to man, counts head lice, and explores every hole in the ground. Gauss, a man born in poverty who will be recognized as the greatest mathematician since Newton, does not even need to leave his home in Gottingen to know that space is curved. He can run prime numbers in his head, cannot imagine a life without women and yet jumps out of bed on his wedding night to jot down a mathematical formula." 
A: Thomas Pynchon's "Against the Day" has several characters who are mathematicians, with some of the action taking place in Goettingen in the early twentieth century. There are many mathematics references and jokes throughout the book which only mathematicians would get.
A: The Housekeeper and the Professor, about a
mathematician whose short-term memory is limited to only 80 minutes.
Reviews: e.g, NY Times, Guardian, Culture Vulture (a blog).
Full of praise for the beauty of math (in particular concerning prime numbers), and of baseball.
A: Kepler by John Banville is a sort of 'fictional biography'.  Banville is a Booker prize winner, very highly regarded.  His prose is some of the most beautiful, dense and lyrical I've ever read, and I'd recommend Kepler to anyone with an interest in mathematics and a taste for masterful writing.
(Banville also wrote Doctor Copernicus, which I haven't read.)
A: "The Peace War" by Vernor Vinge had a rather nice passage about the coming-of-age of a teenage math whiz, self-taught until meeting up with an established mathematician (shades of Ramanujan-Hardy).  Excerpt:

Wili spent a week dreaming
up other orthogonal families and was
disappointed to discover that most of
them were al­ready famous - haar
waves, trig waves - and that others
were special cases of general families
known for more than two hundred years.
He was ready for Naismith's books now.
He dived into them, rushed past the
preliminary chapters, pushed himself
toward the frontier where any new
insights would be beyond the farthest
reach of previous explorers.
In the
outside world, in the fields and the
forest that now were such a small part
of his consciousness, summer moved
into fall. They worked longer hours,
to get what crops remained into
storage before the frosts. ...

While those two are major characters in the book, it doesn't get mathematical like that much more that I remember (it's been quite a few years since I read it).
A: The Last Theorem by Arthur C Clarke and Frederik Pohl (and the late Clarke's final book). The main character is a young Sri Lankan mathematician who isn't satisfied with Wiles' proof of Fermat's Last Theorem. There is a reasonable amount of discussion of mathematics for a mass-market science fiction novel, if I recall correctly.
A: John Banville is perhaps the best stylist I've ever read. His style is poetic, concise (few of his books are much over 200 pages), perhaps even mathematical. Many of his protagonists are mathematicians, but that might not make a great deal of difference to this community as mathematics is usually discussed only in passing. Banville has a lot to say about the creative experience and travails of being a mathematician, but if you're looking for, say, witty insights about topology, best look elsewhere.
Banville's work with the most to say about the general experience of being a mathematician is Mefisto. The protagonist is a mathematician, probably a chaos theorist. Mefisto never goes into the details of his work, but nearly every word is a reflection on the working style of a mathematician and the necessary conditions for great discoveries to occur. The book shares some themes with Aranofsky's movie Pi. Mefisto is loosely based on Faust.
Banville's fictionalized biographies of Copernicus and Kepler contain his greatest scientific detail. Personally, I slightly prefer Doctor Copernicus, though Kepler is also very good, particularly the passages about Kepler's battles with his patron Tycho Brahe over geo-heliocentrism. Doctor Copernicus contains some absolutely beautiful passages about the process of doing science, and the details of Copernicus' life, both real and fictionalized, are fascinating.
The protagonist of The Newton Letter is a historian studying Newton's nervous breakdown. Parallels between Newton's experiences and the historian's begin to emerge.
The protagonist of The Book of Evidence is a statistician turned murderer and art thief. Statistics isn't discussed much, though there are a few snappy lines about what qualities make a good mathematician. I haven't read Ghosts, but, apparently, Ghosts has the same characters as The Book of Evidence.
Banville's most recent book, The Infinities, is about a son's return to the deathbed of his father, a famous mathematical physicist. The book has little to say about math: more to say about celebrity, I think. Pretty much everything mentioned about math is deliberately counterfactual. For example, it's mentioned in passing that cold fusion is a reality, and that the father got famous for his general equations explaining why it works.
The two books I particularly recommend to scientists are Doctor Copernicus and Mefisto.
A: In the short story Division by Zero by Ted Chiang, a mathematician discovers a proof that 1=2. The story discusses (among other things) its effect on her, but other (real-life) mathematicians and their ideas are also mentioned.
A: Written about mathematicians, by mathematicians, and certainly for mathematicians, the self-published "choose your own adventure," Mathematics Odyssey (Wayback Machine), certainly deserves mention.
A: Anathem, by Neal Stephenson.
The book is set in an alternate timeline, where a tradition has taken root (over thousands of years) for academics to isolate themselves from society for long periods of time. There are those who are in for a year, and aren't allowed to mingle/communicate with those who are in for 5 years, or 10, or 100. There are some actual proofs in the text that are crucial to the plot. The book contains a serious discussion (which I wouldn't have thought possible, much less interesting) of whether mathematics is discovered or invented, and what would the nature of an experiment that could distinguish these possibilities be like.
I liked the book when I read it. It has really "stuck to my ribs", however, and I now love the book.
A: See also Complots mathématiques à Princeton, by Claudine Monteil.
A: There is Dis-mois qui tu aimes (je te dirai qui tu hais), which is a murder mystery set in a thinly veiled 1980's version of the IHES mathematical community. The link is to a current discussion of how to get a copy (it has been reprinted under a different title, but under either title is currently out of print).
A: Clifton Fadiman assembled a couple of anthologies of stories featuring mathematics and/or mathematicians as main characters.
Fantasia Mathematica
https://www.amazon.com/Fantasia-Mathematica-Clifton-Fadiman/dp/0387949313/
The Mathematical Magpie
https://www.amazon.com/Mathematical-Magpie-Clifton-Fadiman/dp/038794950X
Some are good, some are not so good.
A: The book "De wilde getallen" by Philibert Schogt ("The wild numbers") is a great story about a young mathematician and his struggle with an (imaginary) theorem in number theory. It illustrates the emotional rollercoaster one sometimes goes through while trying to prove theorems.
A: Whom the Gods Love. The Story of Evariste Galois, 1950. Wydanie polskie Wybrańcy bogów. Powieść o życiu Ewarysta Galois.
I read the Polish version in grade school. It is more of a novel than a biography.
The author: Leopold Infeld, a physicist and collaborator of Einstein
https://en.wikipedia.org/wiki/Leopold_Infeld
A: The above mentioned "The Oxford Murders" and "Measuring the World" are two books people must be warned of. The first is the worst detective story that can be possibly written, the second was slated for very good reason in the Notices (http://www.ams.org/notices/200806/tx080600681p.pdf).
Joel Adler
A: The curious incident of the dog in the nightime
https://www.amazon.com/The-Curious-Incident-Dog-Night-Time/dp/1400032717/ref=sr_1_1?ie=UTF8&qid=1342469323&sr=8-1&keywords=the+incident+of+the+dog
This book got many awards.  Read a review above or at
https://en.wikipedia.org/wiki/The_Curious_Incident
Don DeLillo - Ratner´s star:
https://www.amazon.com/Ratners-Star-Don-Delillo/dp/0679722920/ref=sr_1_1?s=books&ie=UTF8&qid=1342469693&sr=1-1&keywords=Ratners+star
A: The Man Without Qualities (Der Mann ohne Eigenschaften) by  Robert Musil. The protagonist is a Mathematician, and the book runs through an amazing range of issues concerning science, technology and their relations to society and to the nature of consciousness. It is set in the last days of the Austro-Hungarian Empire.  The novel was never finished -- the published fragment runs to around 1700 pages.
A: There is a graphic novel called Logicomix, by Apostolos Doxiadis and Christos Papadimitriou, which is about Bertrand Russell and the search for the foundations of mathematics.  I know you were asking for fiction, but as with Kepler,  this is sort of a fictionalized version of actual events, rather than an academic history book.  Plus the pictures are great!  And it's self-referential; the authors themselves appear in the book.
I liked this one a lot.
A: I'm surprised no one has mentioned The Foundation Series yet, by Isaac Asimov.  The fictional mathematician Hari Seldon even invents the fictional mathematical discipline of psychohistory.
A: Uncle Petros and Goldbach's conjecture is a story about a fictitious mathematician who became obsessed with solving the Goldbach's conjecture. The main theme of the book is obsession and it is quite engaging. It also has the mathematicians Ramanujan, Hardy, Littlewood, Turing, etc. as characters.
A: Few days ago I bought and already read this book, which I found quite nice (sorry it is in Russian):
Тельняшка математика ( Mathematician's (sailor's) striped vest) 
Author: Игорь Дуэль (Igor Duel )
http://prochtenie.ru/index.php/docs/3895
http://www.ozon.ru/context/detail/id/5020607/
This fiction story is about young gifted mathematician. To make the story interesting for (general) audience the authors uses the following tricks: 
1) the results obtained by main hero were attempted to be stolen by his boss, high-ranked
administrative official, who  is very weak in math, but build his career on the works of others. So this increases the temperature of the exposition and hopefully everyone will sympathy the main hero.
2) Main hero having this problem in his career makes a change in his life and goes to work as a sailor on a ship for many months. (That is why the title is so). 
So the exposition organized in the following way: chapter about math-life, chapter about sailor's life. During his sailor's life he has  many of adventures, and meet many different people.
3) The part of process of making discoveries is also described in way that everyone can
try to understand - by the analogy with military compaign: the author's results 
stands on results of his older colleague (springboard for attack), 
he try and fail the "front-wide" "Blitzkrieg", and after that
he creates his new paths to  unknown (enemy's) territory,
in several directions: in forests, in bogs -- in order to find the general picture
from pieces.
4) Of course, there is line  about love story of the main hero.
A: The Oxford Murders is a nice novel about a graduate student in Oxford, which specializes in logics (it seems that the author has a PhD in logics).
The book is kind of nice, and it was adapted to a film a few years ago.
A: John Wallis is one of many 17th century mathematicians and scientists who appear as significant characters in Iain Pears's novel An Instance of the Fingerpost --- a book I highly recommend.  (Warning:  Do not read the Amazon customer reviews.  They're full of major spoilers.)
A: Mathematicians in Love is a book by Rudy Rucker. The protagonist is a graduate student in mathematics, as is his rival. The ideas he is trying to put into his thesis turn out to be the keys to altering reality.
Rudy Rucker has written nonfiction about mathematics, so I expect that many of his other fictional works are influenced by mathematics.
A: The Broken God by David Zindell is a sci-fi novel about a universe in which the top ruling class is called "The Order of Mystic Mathematicians and Other Seekers of the Ineffable Flame". They have to go through extreme training in advanced math because they use topology to navigate through the universe using something called the Vild. To my knowledge it is the only work of fiction that uses the term "topology" in its mathematical sense in the first few pages.
A: After Math by Miriam Webster is a kind of mystery novel set in a department of mathematics.  The main characters are mathematicians and there is a considerable amount of mathematics in the book; I believe that the author (whose real name is Amy Babich) has a Ph.D. in mathematics.
A: I recently read a short story called "Space" by John Buchan (author of The Thirty-Nine Steps) which is about a mathematician who discovers a gateway to the fourth dimension or something. It's quite good.
Also, Charles Kingsley wrote a novel about Hypatia of Alexandria back in the mid-nineteenth century. I haven't read it yet, but I found two copies in a secondhand bookshop the other day. I would be surprised if it didn't have some mathematics in it.
Edit: I just finished reading "Hypatia". Surprisingly, it didn't have any mathematics in it. It did contain this nice quote, however:

In the hour of that unrighteous victory, the Church of Alexandria received a deadly wound. It had admitted and sanctioned those habits of doing evil that good may come, of pious intrigue, and at last of open persecution, which are certain to creep in wheresoever men attempt to set up a merely religious empire, independent of human relationships and civil laws; to 'establish,' in short, a 'theocracy,' and by that very act confess their secret disbelief that God is ruling already.

A: Godel, Escher and Bach: An eternal golden braid?
A: One might argue whether the main character D-503 of Yevgeny Zamyatin's We counts as a mathematician. It is a classical dystopian novel, similar in spirit to Brave New World or Nineteen Eighty-Four (but older and arguably better). D-503 is rather an engineer than a mathematician in our sense, but in the novel's setting, due to the almost complete mechanization of human life, this is as close as one can get to being a mathematician. In any case, mathematical concepts play a decent rôle, in particular - as strange as this may sound - the purported challenge to imagination posed by imaginary numbers.
A: Nobody has yet mentioned "The French Mathematician" by Tom Petsinis (http://amzn.to/NprQMg) a novelized telling of the life of Galois.  As well as being a good read, it's meticulously researched (the author, as well as being an award winning novelist, playwright and poet, is an accomplished mathematics educator.)  I recommend it to you all.
A: Alex`s Adventures in numberland ny Alex Bellos. Here is a link to his website and an interview:  http://alexbellos.com/ https://www.youtube.com/watch?v=ci3P5jf48cY
A: The solitude of prime numbers of Paolo Giordano is a fiction book about a mathematician.
A: A book I like a lot is "Whom the gods love, the story of Evariste Galois" by the physicist Leopold Infeld. It is mostly based on the known facts about his live. Infeld however fictionalized the missing parts. All in all it makes a very readable biography/novel of the originator of group theory and Galois theory written by this collaborator of Einstein and Max Born. Unfortunately the book is not available anymore, and very few university libraries have it.
A: Cryptonomicon is a book by Neal Stephenson. One of the main characters in the WWII timeline is Lawrence Waterhouse, a mathematician and cryptologist. Waterhouse gets to interact with Alan Turing.
Neal Stephenson later wrote a series (the Baroque cycle, starting with Quicksilver) which featured ancestors of the characters in Cryptonomicon. The Waterhouse in Quicksilver works with Isaac Newton and encounters Leibniz, but is not a mathematician.
(I don't think either of these works is as good as Stephenson's Snow Crash, but that doesn't feature mathematicians.)
A: "Proof" is a play, but it is based upon a women finishing a proof that her father was unsuccessfully working on as he slips into mental decline. In fact, the play has done quite well, and they even made a movie version with Gwenneth Paltrow, Anthony Hopkins and Jake Gyllenhall. 
When you mention the audience being narrow and adding other elements to attract a larger spectrum of the population, I would amend your comment about spies and intrigue to note that a majority of the "popular" novels and movies about mathematicians involve mental illness as a "lead" character. Also true in "A Beautiful Mind". One could argue the same for "Flatland," but the interactions of the main character are a little more "antisocial" than "mentally ill," per se.
A: Greg Egan is a science-fiction writer that holds a B.S. in Mathematics (and has co-authored a paper with John Baez). He often manages to insert some advanced maths, physics and computer science content in his novels: for instance, listing only mathematics, fiber bundles in Diaspora, Einstein's equation for general relativity in Incandescence, Cantor sets and commutative hypercubes in the short stories The Infinite Assassin and Glory.
His story Dark Integers (Wayback Machine) deserves special mention; it is a sequel to Luminous, best read in order.
It is truly science fiction written for scientists and mathematicians in particular; they are the only readers that are able to grasp fully both the casual references to advanced mathematical content and the grand ideas underlying his stories. Even after a master in pure maths and a phd in numerical analysis, often I feel that I do not know enough geometry and theoretical physics to get all the facets and implications of what he writes.
This feature sets him apart from most other writers in this list, who address maths from a popular-science point of view.
On the top of my head, I find it difficult to name a novel of his that does not feature a scientist among the protagonists.
A: I recommend The Man Who Counted, an amazing fiction book about a fictitious mathematician called Beremiz Samir and a traveler called Malba Tahan.
The first two chapters tell how Malba Tahan was traveling from Samarra to Baghdad when he met Beremiz Samir, a young lad with amazing mathematical abilities. The traveler then invited Beremiz to come with him to Baghdad, where a man with his abilities will certainly find profitable employment. The rest of the book tells various incidents that befell the two men along the road and in Baghdad. In all those events, Beremiz Samir uses his abilities with calculation like a magic wand to amaze and entertain people, settle disputes, and find wise and just solutions to seemingly unsolvable problems.
In the first incident along their trip (chapter III), Beremiz settles a heated inheritance dispute between three brothers. Their father had left them 35 camels, of which 1/2 (17.5 camels) should go to his eldest son, 1/3 (11.666... camels) to the middle one, and 1/9 (3.888... camels) to the youngest. To solve brother's dilemma, Beremiz convinces Malba to donate his only camel to the dead man's estate. Then, with 36 camels, Beremiz gives 18, 12, and 4 animals to the three heirs, making all of them profit with the new share. Of the remaining two camels, one is returned to Malba, and the other is claimed by Beremiz as his reward.
ADDED:

*

*The Man Who Counted at Amazon

*A nice review at Math Hombre
A: Alex Kasman's "Mathematical Fiction" website
http://kasmana.people.cofc.edu/MATHFICT/
offers an extensive list (with synopses and/or reviews) of books.
A: Someone else mentioned Hari Seldon in Asimov's Foundation Series, but there is also a wonderful Asimov short story in which the protagonist is a government mathematician who discovers, by analyzing primitive computers, how to do computations such as multiplication and division with only paper and pencil. (Actually, he may be writing on a pad, but anyway, he's doing it by hand.)
SPOILER ALERT  (The rest of this post gives away some, but not all, of the plot.)  
The point of the story is that no one on earth remembers how to do even arithmetic, everyone just relies on calculators and computers. The government becomes very interested in this discovery, because it has the potential to allow them to build war rockets piloted by people (inexpensive), rather than by computers (expensive). Given the extent to which children these days are taught to do arithmetic using calculators, Asimov's story is quite prescient. The only problem is that I don't remember the title, maybe someone on MO can help out with that.
A: You could try "Solar" by Ian McEwan. It's about a senior researcher that hasn't done good work in years but gets a break one day.
Somewhat less seriously, there's also the delightfully named "Advanced calculus of murder" by Eric Rosenthal.
A: Rymdväktaren and Nyaga are (admittedly Swedish language) sci-fi books that feature 5-6 mathematicians in the main cast as well as one supercomputer.
A: La conjecture de Syracuse, by Antoine Billot (in French).
This novel is about a fictional mathematician having solved the Collatz conjecture in his youth and whose career will be put at risk by a young student competing with him.
A: The protagonist of Euclid's Alone by William Orr, and that of Four Brands of Impossible by Norman Kagan. (both short stories can be found in the collection "Mathenauts" http://kasmana.people.cofc.edu/MATHFICT/mfview.php?callnumber=mf52 )
A: Not a book, but a play: "Descartes and Pascal", about a historical encounter of these two mathematicians. It was in part translated into Polish and  I remember reading it in this version. I could not recall the name of the author. Googling gives Jean De Clouse, by a reference to a stage version in Marathi:
Written by Jean De Clouse and translated into Marathi by Madhuri Purandare, the play's subject derives from a historical meeting between philosopher-mathematician Rene Descartes and Blaise Pascal, a mathematician, physicist and philosopher himself. This meeting took place on 23rd September 1646.
If someone could confirm or correct the authorship or/and the title, he/she is welcome. There are other  references to the meeting on the web, but not to the play:
https://www.independent.co.uk/arts-entertainment/when-blaise-pascal-met-rene-descartes-1337242.html
A: "Bimbos of the Death Sun" and "Zombies of the Gene Pool" by Sharyn McCrumb feature an engineering professor who writes science fiction and solves murders.  Two of the funniest books I've read.
A: According to Wikipedia, https://en.wikipedia.org/wiki/Giordano_Bruno, Giordano Bruno was a mathematician, among other things, and S J Parris has written three excellent novels (and counting ..) that feature Bruno as the main character as a detective and secret agent combined:
https://www.amazon.com/Heresy-S-J-Parris/dp/0385531281
https://www.amazon.co.uk/Prophecy-Giordano-Bruno-S-Parris/dp/0007317735
https://www.amazon.co.uk/Sacrilege-Giordano-Bruno-S-Parris/dp/000731776X
A: This might qualify as experimental mathematics fiction.
The Nine Billion Names of God by Arthur C. Clarke.

monks created an alphabet in which they calculated they could encode all the possible names of god, numbering about 9,000,000,000 ("nine billion") and each having no more than nine characters. Writing the names out by hand, as they had been doing, even after eliminating various nonsense combinations, would take another 15,000 years; the monks wish to use modern technology in order to finish this task more quickly.

A: Limiting myself to books not yet mentioned:
Einstein's Dreams by Alan Lightman is notable for luminous prose. It does pertain nominally to physics, not pure math, but only (at the time) of the most theoretical kind, far closer to math than most physics of the day.
As to the Foundation series, which has been mentioned here before since Hari Seldon is a mathematician, it is worth noting that in the later novels of the first trilogy key roles are played by a community of mathematicians whose mathematical research is critical to the advancement of the plot. 
The French Mathematician by Tom Petsinis is a fictionalized account of Galois; I enjoyed it.
Also, Engima by Robert Harris is loosely based on Turing's work in the war. It is not entirely without points of interest. Harris is a pretty good writer, although the story is perhaps not his best.
A: Gillian Bradshaw's "The Sand Reckoner" is a wonderful story about Archimedes's return to Sicily from Egypt.  His work as an engineer building super-catapults is featured more than any mathematical enterprise.  Does that qualify?
A: I was a bit disappointed by Iain M. Banks' The Algebraist. Apart from the nice title, which is what made me buy the book, there is very little mathematical content in it.
(Well, I guess I am spoiled by Greg Egan.)
A: *

*Arno Schmidt's characters sometimes are occupied with mathematics, although mostly as a tool for geodesy or astronomy. A notable instance of pure mathematics occurs in the novelette Schwarze Spiegel (1951): The narrator, believing he is the last man on earth after a nuclear world war, kills time by "proving" Fermat's last theorem -- he even claims that the proof generalises to a proof of Euler's conjecture. Not surprisingly, the "proof", which consists of elementary manipulations, is flawed in several ways -- what it does give is a distorted version of Euclid's formula for Pythagorean triples. (It is not clear to me whether the narrator actually believes in his proof.)


*A chaos theorist is a main protagonist of Michael Crichton's Jurassic Park (1990), and a popularised version of chaos theory serves as an idea in the plot. Less so in the movie Spielberg made of it.


*One chapter in László Krasznahorkai's From North a Hill, from South a Lake, from East a Road, from West a River (2003) consists of a description of a fictitious work The infinite, a mistake, written by one (little less fictitious) "Sir Wilford Stanley Gilmore". It is said that the book's imprint mentions "a small town called Bures-sur-Yvette and the Gilmore-Grothendieck-Nelson-Institut des Hautes Etudes Scientifiques". I will not spoil any more except that it ends with a rant against Georg Cantor. Update: Nearly ten years after I recommended it, this novel has finally appeared in English translation, with the title A Mountain to the North, a Lake to the South, Paths to the West, a River to the East . I still recommend it.
