I distinctly remember that during the early 1990's, there was some kind of prediction market where one of the items was whether Fermat's Last Theorem would be proved before some date (the year 2000 maybe?), as judged by a panel of mathematicians or something.  Unfortunately I do not remember the details; I think that there were periodic postings to sci.math giving the current "going rate," and of course as confidence in the correctness of Wiles and Wiles-Taylor grew, the market gradually dried up.  I haven't been able to confirm my memory with a Google Groups search, though.

Intrade.com does have a few science-related markets but I don't know if they have ever had mathematics-related markets.