**Talk** to the AMS editors directly. I have a late (and dear) coauthor who has published many books and I published one with him. His experience was that scholarly society publishers are very open minded with regards to your copy-left/right needs. Basically, you'll be dealing with other scholars in mathematics who have published works, teach at a university ... and not middle management in a publishing house with an MBA degree (with all due respect to my students). 

My experience with the AMS regarding copyright issues is very positive, though things are administratively chaotic (no updates on sales, no money--though I know for a fact that my mom bought six copies).