To address the issue of Fermat's Last Theorem: the reasoning behind Fermat's Last Theorem, while elaborate, in the end rests on basic intuition about the integers. (I'm not sure that it is actually proved in first order Peano arithmetic, since the proof as written certainly uses concepts outside of PA, but nevertheless, it is basically a result about numbers, proved using our fundamental notions about numbers.)
If the proof was correct, but the statement wrong (due to an inconsistency), there would be something fundamentally wrong in our conception of numbers. I don't think this would be like the crisis in set-theory: it would be much more fundamental. For example, if induction turns out to be inconsistent (and this is the kind of thing being speculated about here), this says that our basic intuition for the natural numbers, namely that non-empty subsets have least elements, is wrong. If that is true, then all mathematics goes out the window!
I think that most mathematicians (indeed, most humans who have been taught arithmetic) have a mental model of the natural numbers which says that you can always add 1 to get a new number, and that between any two natural numbers there are only finitely many more (so that any non-empty subset of the naturals has a least element). Given this, they know that PA is in fact consistent, even though PA doesn't prove this. They are proving it by exhibiting a (mental) model; they don't need formal arguments. (This falls under the class of "not completely formal" methods alluded to by Carl Mummert.)