It seems as though many consider ZF to be the foundational set of axioms for all of mathematics (or at least, a crucial part of the foundations); when a theorem is found to be independent of ZF, it's generally accepted that there will never be a proof of the theorem one way or another. My question is, why is this? It seems as though ZF is flawed in a number of ways, since propositions like the axiom of choice and the continuum hypothesis are independent of it. Shouldn't our axioms of set theory be able to give us firm answers to questions like these? Yes, the incompleteness theorems say that we'll never develop a perfect set of axioms, and many of the theorems independent of our axioms will probably be quite interesting, but is ZF really the best we can do? Is there hard evidence that ZF is the "best" set theory we can come up with, or is it merely a philosophical argument that ZF is what set theory "should" look like?
Very few mathematicions these days wish to base their mathematics on ZF without the axiom of choice, as your question seems to imply. Yes, there is an intuitionist school about which I don't know a whole lot, but they seem to be quite the minority. So let's consider ZFC. I think the main philosophical argument for it is that the axioms seem obviuosly true, if you are willing to believe that such things as infinite sets exist in some fashion, and that nobody has been able to come up with an equally obviously “true” statement about sets that is not a consequence of these axioms. The independence of the continuum hypothesis doesn't seem to bother people much, since to most of us it doesn't seem either obviously true or obviously false. Though some people are working on settling it by finding other axioms that will at least be considered likely true or at least useful. There was a recent article in the Notices about these efforts, in fact. But is there “hard evidence” that ZFC is the best we can come up with? Not by most mathematicians' standards I think, but there seems to be plenty of soft evidence. I might add that many working mathematicians (I am talking here mostly about analysts, since they are the people I know best) don't care one whit about these questions, but happily go about their business using Zorn's lemma whenever it seems necessary and never let it bother them. And there are some who would rather base all mathematics on category theory rather than ZFC set theory, but that is not my cup of tea, so I will leave it unstirred. 


There is a very active ongoing debate within set theory about whether mathematics needs new axioms, and philosophers of mathematics are weighing in on all sides. Relevant considerations include many very deep topics in set theory, including independence, forcing and the large cardinal hierarchy. Some of these topics are at once highly technical and philosophical at the same time. It is fair to say that there is an emerging field called the philosophy of set theory that is grappling precisely with these issues. Let me try to mention just a few of the considerations. First, the historical fact remains that the ZFC axioms are sufficiently powerful to carry out almost all of the construction methods that arise in mathematics outside set theory. Indeed, the ZFC axioms are provably far more powerful than necessary for the vast majority of ordinary mathematics. This is proved by the stunning results of the field of Reverse Mathematics (see Steve Simpson, Harvey Friedman etc.), which calculates for a huge collection of classical mathematical theorems exactly which axioms are needed to prove them. Reverse Mathematics proceeds by proving the axioms from the theorem as well as the theorem from the axioms (over a very weak base theory), thereby showing the necessity of those axioms, and it turns out that most all of the classical theorems of mathematics can be proved in relatively weak theories. Nevertheless, within set theory, set theorists have discovered the ubiquitous independence phenomenon, by which an enormous number of settheoretical assertions turn out to be independent of the ZFC axioms. This means that they are neither provable nor refutable in ZFC. We now have thousands of instances of fundamental set theoretic propositions that are known to be independent of ZFC. This includes almost any nontrivial statement of infinite cardinal arithmetic (such as the Continuum Hypothesis), as well as an enormous number of statements in infinite combinatorics, and so on. This phenomenon supports the view that ZFC is a weak theory, unable to decide these questions. But of course, by the Incompleteness Theorem we know that any theory we can write down will exhibit this independence phenomenon. It is impossible in principle to avoid it. Large cardinals are strong axioms of infinity, some of which go back to the time of Cantor (so they are not new), which are not provable in ZFC and which transcend ZFC in consistency strength, forming a vast hierarchy of consistency strength above it. Thus, they tend to make up for the weakness of ZFC (although there remains extensive independence even with large cardinals). Some set theorists make the case that the existence of large cardinals have numerous attractive regularity consequences, even for low down for sets of reals, that they seem to point the way towards the finally true set theory, which must remain elusively hidden from us because of the Incompleteness theorem. Making sense (or nonsense) of this view is a central concern of the emerging Philosophy of Set Theory. 


People use whatever is most useful. ZFC just happens to be a fairly simple formalization of the way people think of sets such that we can eliminate imprecision sufficiently to do good mathematics. There are no "theorems" independent of ZFC in ZFC. CH is not a theorem in ZFC. Choice isn't either, but on the other hand choice is useful so everyone uses it. If you don't accept the axiom of choice then you can't have things like arbitrary products, and for some strange reason studying infinite products is actually very useful to practicing mathematicians. That's not to say that studying things like CH isn't useful. I think it is, but right now it's just not pertinent to most mathematicians. 


This answer is essentially a Joel's version by another route. ZF(C), possibly with appropriate large cardinal axioms, is one of the three most important formal axiomatisations in the foundations of mathematics, because it is foundationally complete (Friedman 1997):
...



There is Freiling's axiom of symmetry (AX) it is equivalent to the negation of the continuum hypothesis more on this is here: http://en.wikipedia.org/wiki/Freiling%27s_axiom_of_symmetry For a categorical foundation of mathematics look here: 


ZFC is only "standard" because it (or conservative extensions of it like Universes) allows us to do pretty much whatever we want. However, recent developments in higher category theory have led some to call for a new "set theory" (mentioned above) that categorifies the classical theory of sets and cleanses the "evil" from it. (Evil, of course, in the sense of considering objects up to equality rather than isomorphism.) It seems unlikely, to me at least, that someone will be able to develop a useful alternative set theory that isn't equivalent to or stronger than ZFC, the operative word there being useful. That is, we don't want whole edifices of mathematical thought falling into the ocean under our new set theory. 


I think, the worst thing that can happen to mathematics is to impose some dogma onto it. And the worst kind of dogmas are those concerning foundations. It is in the essence of mathematics that an object has its existence justified as soon as it is clear that this object is welldefined. And that´s really all about it. There is no usefulness or appropriateness or 'better' when it comes down to laying the foundations. For axioms are not subordinated to their implications. As for the so called 'intuition', the latter tells me that $\mathbb{Q}$ has greater cardinality than $\mathbb{N}$... In other words I don´t see a single reason why we shouldn´t have 'multiple' mathematics, each based on its foundations, each equally justified! 


I am a bit uncomfortable with your wording; you say “it seems that ZF is flawed” or “ZF is the "best" set theory we can come up with?”… I don’t think that the words “flawed” and “best” are suitable… Mathematics is what mathematicians do, just as Art is what artists do and there are lots of mathematicians doing lots of mathematics. Some are interested in ZF others in Elliptic curves. Considering ZF as a foundation for “all of mathematics” is like considering music as a foundation of “all of art” or Physics as a foundation of “all of science”… Mathematics, Art, Science, Philosophy… have no foundation or if you prefer studying foundation*s* is just one of the many branches of a “discipline” one of many things done by human beings for pleasure. The fact that axiom of choice is independent of ZF and the fact that one might want to consider this axiom as true is just a very interesting question about the hidden puzzles of choice and infinity precisely sort of puzzles that stimulate, motivate and inspire those who like to study them. Wanting to have a framework that settles this precise question might excite some but bore others. Personally I think that event the “finite version of the axiom choice” needs thought… 

