I run a web site, theassayer.org, which catalogs books whose authors have intentionally made them free. A surprisingly high percentage of such books are advanced math books, the other large category being computer manuals and computer science books. If you browse through the catalog, you can get an idea of what various people are doing: what licenses they're using, whether they sell the book in print, whether they're working with a traditional publisher or self-publishing.

My own experience is that \$20 is too high for my students. At that price, nearly all of them would prefer the free download. Students are cheap, and they will put up with amazing hassles to save a buck. As time has gone on, and it's become more and more common for students to carry various electronic devices with them, the maximum price they'll pay has dropped lower and lower. I currently sell the printed copies of my books through lulu, with zero markup so that I receive no revenue. This makes the price about \$12 for a 500-page paperback that they use for one semester. Even at that price, only about half will pay for a printed copy. The other half use the free download, often on a phone with a screen the size of a postage stamp.