I think your reading is wrong.  Set theorists have studied all sorts of additional axioms, some implying CH, some being strictly weaker than CH, and many contradicting CH.  My understanding is that most set theorists today, if they have any opinion on the matter, prefer to think that CH is false.  In particular, Woodin has recently advanced a philosophical argument for a strong additional axiom of set theory that implies that the continuum is \aleph_2.

However, the fact of the matter is that CH has very little effect on "ordinary mathematics".  You can come up with a few down-to-earth seeming combinatorial statements that are equivalent to CH, but it really just never comes up if you never deal with objects that are either very infinite or are infinite and don't have a lot of structure attached to them.  For reference, it is consistent for the cardinality of the real numbers to be almost any uncountable cardinality (the only requirement is that it have uncountable [cofinality][1].

I don't have time now, but if you want more references on this I can try and add some later.


  [1]: http://en.wikipedia.org/wiki/Cofinality