Though I haven't dealt directly with them, I'm aware of another established company (in Ohio) which buys and sells advanced or rare books in mathematics: http://www.zubalbooks.com/index.jsp
Except for purely local transactions, shipping cost is always a major concern in dealing with individual books or small collections (more so outside the US). But the market for advanced mathematics is limited everywhere, so be selective. It's true that most public or college libraries have too little shelf space and staff to deal with questionable freebies. I've often given away surplus books at all levels to colleagues and students, but there is no way to guarantee that these are really used. Some I've given away have on the other hand wound up being sold, as I later learned.
People stop by faculty offices here regularly and offer cash for current sellable editions of elementary textbooks; they pay well but are definitely picky. Even that market is changing rapidly due to e-books and the like.
