CC itself has advice for authors about this: 

* http://creativecommons.org/weblog/entry/13694

O'Reilly hasn't done math books (yet), but they have lots of computer books under CC licenses:

* http://oreilly.com/openbook/