Where to publish a math textbook in Creative Commons The question I am asking concerns publishing and not mathematics in a strict sense, the argument might be off-topic for this site... but I will try anyway.
Suppose that you are writing a textbook in advanced mathematics, say a book for mature undergraduates or graduate students. Suppose that: 


*

*You write everything using Creative commons, possibly in its most liberal version, which allows people to copy and rewrite your text, and maybe even to use it for commercial purposes!

*As a consequence of the CC license, your text will certainly be freely available on the web. Nevertheless, you want to publish the book at some reasonable price, say 20$. You presume that a student is willing to pay a reasonable price for a hard copy even if it is freely available on the net.

What would you do? Is there any standard publisher in the mathematical community who would publish your book?

Further on, you might also realise that you are doing all the job (writing and formatting), and that you actually don't really need a publisher at all ... would you consider self-publishing?
 A: Cambridge University Press is perfectly willing to publish books which are also freely available on the web (at least in 99%-complete draft form).  I'm not sure how far they'd go in terms of the most liberal Creative Commons license, but here are a couple of examples of it occurring:
http://theory.stanford.edu/~tim/ -- scroll down to find the link to "The AGT Book".
http://www.cs.princeton.edu/theory/complexity/
I think there are additional other examples from Cambridge; those are just two I knew off the top of my head.
Edited to add:  Sorry, I didn't notice that David Speyer already pointed out Cambridge University Press in the context of Allen Hatcher's Algebraic Topology book.
A: You might try contacting Lon Mitchell at Virginia Commonwealth University.  VCU has published hardcover versions of two open source textbooks (one on Linear Algebra, the other on Abstract Algebra), each costing under $20.
A: I run a web site, theassayer.org, which catalogs books whose authors have intentionally made them free. A surprisingly high percentage of such books are advanced math books, the other large category being computer manuals and computer science books. If you browse through the catalog, you can get an idea of what various people are doing: what licenses they're using, whether they sell the book in print, whether they're working with a traditional publisher or self-publishing.
My own experience is that \$20 is too high for my students. At that price, nearly all of them would prefer the free download. Students are cheap, and they will put up with amazing hassles to save a buck. As time has gone on, and it's become more and more common for students to carry various electronic devices with them, the maximum price they'll pay has dropped lower and lower. I currently sell the printed copies of my books through lulu, with zero markup so that I receive no revenue. This makes the price about \$12 for a 500-page paperback that they use for one semester. Even at that price, only about half will pay for a printed copy. The other half use the free download, often on a phone with a screen the size of a postage stamp.
A: Have you tried with Amazon's service BookSurge? You might find it at
http://www.booksurge.com/
I cite this because J. S. Milne (also on MO whose website is http://www.jmilne.org/math/index.html) 
 published two books (one on Elliptic Curves, the other is a new edition of his very famous Arithmetic Duality Theorems) and prizes are extremely cheap for both. Nonetheless, I cannot saying anything on royalties, as I do not know, and heard some rumors that Milne himself was not so happy about something, but it may very well be false: why don't you contact him?
A: CC itself has advice for authors about this: 


*

*http://creativecommons.org/weblog/entry/13694
O'Reilly hasn't done math books (yet), but they have lots of computer books under CC licenses:


*

*http://oreilly.com/openbook/
A: There seem good advantages in POD (Print On Demand) publishing. I discuss some issues in POD   at http://pages.bangor.ac.uk/~mas010/orderbook.html
I have found createspace a good publisher for my book "Topology and Groupoids"; they are an amazon.com company, and this has the implication that the royalty rates are good (black and white, 6" x 9", 200 pages,   price  20.00 dollars, gives $8.75 royalty). Non exclusive contract. 
You should consult the book "The Fine Print", written by a copyright lawyer, Mark Levine.  
All the publicity, and sending out review and complimentary copies,  has to be done by you - this is a typical downside to any self publishing. 
Later: Another disadvantage is that many libraries and institutes buy anything issued by a major publisher, but have to be persuaded to buy a book which does not have the cachet of having satisfied the refereeing process of a major publisher. The advantage of this  amazon.com company is that the royalty is direct on the published price for amazon sales.  
Later: I have found another problem, as a UK resident, namely that the USA Inland Revenue insists that 30% tax be deducted, as against the lowest UK rate of 20%, unless you can get a USA Income Tax Number allocated: in view of the difficulties I have had in doing this, I think you would need to employ a professional who knows all the ropes. However, this is a marginal problem in my case. 
A: I hesitate to call this an answer, since it is based on no actual experience, but I would make three points:
(1) It is not clear to me that you have no use for a publisher. Publishers work to promote your book to libraries, book stores and to faculty looking for a textbook. They also may be willing to pay you an advance on your future royalties, which would opportunity risk of missing other forms of income by writing a textbook. I also would not assume that book publishers would not provide useful editing if you wanted it. You might be interested in Cory Doctorow's thoughts on the benefits of working with traditional publishers. Doctorow is a fiction writer and essayist who distributes all of his work under a very permissive CC license. Fiction isn't textbooks, of course, but I think there are some interesting ideas here.
(2) If your goal is to print copies of a PDF file with no professional editing, Lulu has a good reputation for that sort of thing. I have no personal experience with them, and the people I do know who have experience were printing fiction and prose, not math, so take that for what it is worth.
(3) The obvious person for you to talk to Allen Hatcher. His Algebraic Topology textbook is freely available for download, but is also sold in hard copy by Cambridge University Press (and is very good!).
A: In addition you might like to consider disseminating your book through the textbook platform Flooved, free content with annotation and other functionalities: http://www.flooved.com/
Already a nice growing collection.
A: Talk to the AMS editors directly. I have a late (and dear) coauthor who has published many books and I published one with him. His experience was that scholarly society publishers are very open minded with regards to your copy-left/right needs. Basically, you'll be dealing with other scholars in mathematics who have published works, teach at a university ... and not middle management in a publishing house with an MBA degree (with all due respect to my students). 
My experience with the AMS regarding copyright issues is very positive, though things are administratively chaotic (no updates on sales, no money--though I know for a fact that my mom bought six copies). 
A: Wikibooks and Wikiversity host textbooks and lectures, respectively, licensed under Creative Commons Attribution-ShareAlike.
They are hosted by Wikimedia Foundation, the same non-profit that hosts Wikipedia and other sister projects.
Books and lectures can be downloaded in printed form or ordered online from PediaPress (which WMF has a partnership with) or a third party.
