Why might André Weil have named Carl Ludwig Siegel the greatest mathematician of the 20th century? According to Steven Krantz's Mathematical Apocrypha (pg. 186):

As was custom, Weil often attended tea
  at [Princeton] University . Graduate
  student Steven Weintrab one day went
  about the room asking various famous
  mathematicians who was the greatest
  mathematician of the twentieth
  century. When he asked Weil, the
  answer (without hesitation) was "Carl
  Ludwig Siegel (1896-1981)."

As the title of Krantz's book suggests, the anecdote may be apocryphal. However, there are other better grounded accounts of great mathematicians expressing the highest admiration for Siegel:
(A) In The Map of My Life Shimura wrote:

I always thought that few people
  really understood my work. I knew that
  Chevalley, Eichler, Siegel, and Weil
  understood my work, and that was
  enough for me [...] Of course [Siegel]
  established himself as one of the
  giants in the history of mathematics
  long ago [...] Among his
  contemporaries, [Weil] thought highly
  of Siegel [...]

(B) In an published interview (pg. 30) Selberg said

[Siegel] was in some ways, perhaps,
  the most impressive mathematician I
  have met. I would say, in a way,
  devestatingly so. The things that
  Siegel tended to do were usually
  things that seemed impossible. Also
  after they were done, they seemed
  still almost impossible.

Why might Weil, Shimura and Selberg have been so impressed by Siegel? I should emphasize that I'm not trying to precipitate a debate about the relative standing of historical mathematicians - rather - I'm hoping to learn about aspects of Siegel's work that I might otherwise overlook. I'm also not looking for, e.g. quotations from the Wikipedia article on him, but rather, less familiar material.
 A: Recently I was digging through the meeting minutes for IAS and chanced upon a report prepared for Siegel to justify hiring him. The relevant pages are 13-18. In this report, it is stated that "we decided to propose at this juncture only one name, because it eclipses all others - that of Carl L. Siegel". 
Later on, an excerpt from a letter from Courant is given which states "the only one of his generation whose strength could be compared with that of the mathematical heroes of the preceding era". More praise is lavished on by Chevalley, where he views Siegel as "on a level with a Hilbert or a Poincare" and by Hardy where he claims that nobody questioned if "he was the equal of any mathematician in his generation, and certainly I never doubted it myself". 
With this in mind, it seems that it was in fact the consensus of the mathematical community then that Siegel was the greatest of his time, not just of Weil. Details of his accomplishments are given in the report for those interested. One thing I personally find puzzling is the lack of his influence in current areas of modern mathematics, as suppose to say Weil or Kolmogorov. 
A: On top of all the classical work that has been mentioned, Siegel made important contributions to other areas, and eventually became a major figure in Dynamical Systems. It is not just anyone who can switch fields and remain at the top. See my account of his proof of existence of Siegel disks.
A: I would just like to mention that in addition to the many shared research interests of Siegel and Weil, they were clearly well acquainted with one another, with both attending Max Dehn's seminar on the history of mathematics over a period of years.  From pages 124-127 of Yandell's book "The Honors Class" and pages 51-53 of Weil's "The Apprenticeship of a Mathematician", it shows that Siegel attended the seminar regularly from 1921-1935 while Weil began attending in 1926 and during subsequent visits to Frankfurt says "as often as I could".  So Weil's high esteem for Siegel is doubtless influenced by personal interaction at the seminar (and elsewhere) in addition to familiarity with his research.
A: In addition to the other insightful and informative answers, for veracity it probably should be noted that there is a significant chance that Weil's remark about Siegel was disingenuous or sarcastic, with some ulterior motivations, all the better that it is defensible, and perhaps out-of-the-blue to listeners at the time.
Being somewhat acquainted with many of the bigger-than-life persons mentioned in other answers, and while greatly respecting their work, I would not have much hope of getting a straightforward, sincere answer from any of them about any questions that touched their own accomplishment and potential place in the historical record, or even touched a question of their taste.
The defensibility on scientific grounds of nominating Siegel at mid-century for the greatest mathematician in that century is a little misleading, also, given the non-smoothness of mathematical activity (what with wars and such getting in the way, too). 
In addition to complicated sarcasm, there is a large possibility that Weil chose that moment to attempt to invalidate other parvenues' claims to "kewlness", by referring to an ur-classical figure. 
(Another point, referring to another answer: it is my impression that Weil would not have thought his "Foundations" (of alg geom) was a long-lasting edifice, but, rather, that it was a stop-gap measure. The point at the time was that the "geometric Italian school" had not provided proofs of a certain sort... and that that issue had produced false conclusions, not only that the proofs/heuristics were not clearly airtight. It was a different time. There was no definition of "Jacobian" in positive characteristic. Arguments "by continuity" that had plausible sense, if still unrigorous, in characteristic $0$ had dubious sense in positive characteristic. Indeed, in 1970, say, as I can personally aver, it was certainly not the case that all the world had capitulated to Grothendieck's idea of alg geom.)
But Pete Clark's essay is more constructive, less dragged down by issues of personality, ego, prejudice, arrogance! :)
Nevertheless, one should reserve endless discounting for the effects of personality, ego, prejudice, arrogance on questions of taste or judgement. :)
... exemplified in the silly-but-profoundly-explanatory "Why can't a woman ... be more like a man?"
A: Apparently, the respect was not reciprocated. I heard a story about Siegel (near the end of his life) asking Paul Cohen (who was no fan of Weil, since they had intersected at Chicago in the fifties): So what happened to that promising young man Andre Weil? He wrote a nice thesis, but what happened since then?
A: No one with any familiarity with his work can doubt that Siegel was one of the greatest mathematicians of the 20th century.  Weil was a decisive, opinionated man -- just the type of person who would have an answer to this question ready at hand.  And "Carl Ludwig Siegel" is a totally unsurprising answer from anyone.  (Also "Andre Weil" would be a totally unsurprising answer from anyone: it might be my answer!)
But it is especially unsurprising coming from Weil.  The list of contemporary mathematicians of the Siegel-Weil caliber is short enough, and among mathematicians on that list -- e.g. Wiener, von Neumann, Kolmogorov, Godel -- 
the research interests of Siegel and Weil were especially close: for instance, there is a Siegel-Weil formula.  Both brought their prodigious knowledge and technique to bear on number theory, but with distinct, and distinctive, styles.  To be very brief and crude, Weil had a fundamentally algebraic approach, whereas Siegel had a fundamentally analytic approach.  My own approach to mathematics is rather close to Weil's (although in magnitude, microscopic compared to his): I very much appreciate that finding the right bit of "structure" can make the solution of your problems self-evident.  A lot -- by no means all -- of Weil's work is like that: the finished product is so tidy and efficacious that you too easily forget to ask how he thought of any of it in the first place.  To someone with this "algebraic" style, Siegel's work looks like a sequence of miracles.  So it is unsurprising to me that someone like Weil would select someone like Siegel to give his top regards.
I think you can also gain some insight into why Weil named Siegel by considering their ages: Siegel (born in 1896) was ten years older than Weil (born in 1906).  Ten years is long enough for Siegel always to have been ahead of Weil in his career and stature, but short enough for them to be true contemporaries and competitors.  Most other great mathematicians that spring to mind when I think of Weil are actually quite a bit younger, e.g. Serre (born 1926), Tate (born 1925), Shimura (born 1930); it makes sense that Weil is not going to name any of these as the greatest mathematician of the 20th century.  Indeed all three are alive well into the 21st century.  
[Added: I just remembered that Chevalley (born 1909) was a contemporary of Weil of a similar stature.  But Chevalley was very close to Weil, both personally and in mathematical styles and tastes.  It is psychologically natural to esteem (and fear) most that which is most different from ourselves, not that which is most similar.  Anyway, for Weil to name Chevalley would have sounded arrogant, as if not being able to name himself he picked the person standing right next to him.]  
By the way, I think that Shimura and Siegel are quite similar in style as well as stature.  I read Shimura's autobiography, and I think he is right to be profoundly disappointed that Siegel did not take more of an interest in his work.  Shimura's work is closer to being a continuation of Siegel's (including a continuation of the brilliance, creativity and orginality!) than any other mathematician I can think of, so it is natural that Shimura holds Siegel in high regard.  
There is also something "organic" in the work of both Siegel and Shimura which naturally bristles a bit at the "Bourbakistic" influence of the French school: it seems clear enough, for instance, that the modern theory of "Shimura varieties" is both an addition and a subtraction from what Shimura himself intended.  I know several of Shimura's students, and though they work in what the rest of the mathematical world thinks of as parts of algebraic number theory and arithmetic geometry, in the way they actually think about mathematics they take a more analytic approach...like Siegel.  I have even fewer credentials to speak for Selberg than I do for any of these others, but I imagine that he may have felt a similar kinship to Siegel, i.e., the use of an "analytic" approach to studying problems that others regard as being more algebraic.  
A: For Weil, the Mordell-Weil theorem; for Siegel, the theorem on integral points on curves (genus at least 1). Think about the use of abelian varieties here. Mordell-Weil is sort of about making Mordell's descent work with a general abelian variety rather than just a one-dimensional one. Siegel's result has to go through Jacobians but brings a sharper version of Thue's work on diophantine approximation to bear. These are both "diophantine geometry" today but Siegel's work still looks like a major advance.
Edit: Maybe I left out the point. Didn't Hadamard tell Weil that he should prove the Mordell conjecture, to do a proper job? And the chronology: both those results came out in 1929.
