How dangerous are set-size assumptions? Many students' first introduction to the difference between classes and sets is in category theory, where we learn that some categories (such as the category of all sets) are class-sized but not set-sized.  After working with such structures, we discover that it is still worthwhile to sometimes treat them as if they were set-sized.  But we can't always do so, without running into contradictions.  A natural solution to this problem is the "Axiom of Grothendieck Universes".  We then work not with the category of all sets, but the category of sets inside some given universe.  This type of framework has been used by many notable mathematicians and seems very ubiquitous.  For instance, it was an assumption in my universal algebra textbook.  I would venture to say that most modern set theorists look at such an axiom as a very tame assumption.
My first question concerns the possible dangers of such an assumption, which are not often raised when first learning about this axiom.  One obvious danger is that this axiom might lead to an inconsistency.  In other words, ZFC might be consistent but ZFC+Universes might be inconsistent.  I don't personally subscribe to this belief, but it is certainly a possibility (without further, even stronger, assumptions).
What really concerns me is the possibility of another danger: Could such a system proves false things about the natural numbers?  In other words, even if we assume that ZFC+Universes is consistent, could it be the case that it proves false arithmetic statements?
The motivation for this question came from reading some of the work of Nik Weaver at this link, which argues for a conceptualist stance on mathematics.  In particular, if we reject the axiom of power set, we are led to situations where all power sets of infinite sets are class-sized.  Nik puts forth the idea that ZFC might prove false things about the natural numbers.  Is this a real possibility?  I suppose so, since it is possible that ZFC is inconsistent but the natural numbers aren't.  But is it still possible even if ZFC is consistent?  More generally:

Could treating the power set of the natural numbers as a set-sized object, rather than a class, force us to conclude false arithmetic statements (even if such a system is consistent)?

Even if the answer to this question is yes, I'm having a hard time seeing how we could recognize this fact, since according to the answers to this linked question it is difficult to state precisely what we mean by the natural numbers.
A second motivation for this question comes from what I've read about the multiverse view of set-theory.  When creating a (transitive) model $M$ of ZFC, the set $P(\mathbb{N})$ can often be thought of as some "bigger" power set of the natural numbers intersected with the model.  Moreover, via forcing, it seems that one can (always?) enlarge the power set of $\mathbb{N}$.  This does seem to suggest that $P(\mathbb{N})$ is not completely captured in any model.  Thus, in its entirety, perhaps $P(\mathbb{N})$ should be treated as a class-sized object.

Added: I believe that the current proof we have of Fermat's Last Theorem uses the existence of a(t least one) Grothendieck universe.  However, my understanding is that this dependence can be completely removed due to the fact that Fermat's Last Theorem has small quantifier complexity.  I imagine that proofs of statements with higher quantifier complexity that use Grothendieck universes, do not necessarily have a way of removing their dependence on said universes.  How would we tell if such arithmetic statements are true of the natural numbers?  [Some of this was incorrect, as pointed out by David Roberts and Timothy Chow.]

2nd addition: There are some theories that we believe prove false arithmetic statements.  Assuming the natural numbers can consistently exist (which we do!), then both PA+Con(PA) and PA+$\neg$Con(PA) are consistent, but the second theory proves the false arithmetic sentence $\neg$Con(PA).
My question then might be rephrased as:

What principles lead us to believe that "Universes" is a safe assumption, whereas "$\neg$Con(PA)" is not safe, regarding what we believe is "true" arithmetic?  (Next, repeat this question regarding the axiom of power set.)  Is any theory that interprets PA "safe", as long as it is consistent with PA, and PA+Con(PA), and any such natural extension of these ideas?

Another way of putting this might be as follows:

Is the assumption Con(PA) a philosophical one, and not a mathematical one?

This ties into my previous question that I linked to, about describing the "real" natural numbers.
 A: There's a lot going on in this question; let's break it down:

Is the assumption Con(PA) a philosophical one, and not a mathematical one?

I suppose it depends on what you mean by "the assumption Con(PA)":


*

*If you're writing a mathematical proof, then either the foundations you're assuming prove Con(PA) or they don't, and it's a mathematical question which is the case. You'll be mathematically justified in assuming Con(PA) in the former case. In the latter case, there are contexts where "assuming Con(PA)" is mathematically justified -- for example, if you're trying to prove a statement of the form $Con(PA) \Rightarrow P$ for some $P$, then you're allowed to argue by assuming Con(PA) and proving $P$. But probably that's not what you mean.

*If you're just asserting "PA is consistent" in a non-mathematical context, then your statement requires some philosophical unpacking to make precise, and moreover to determine to what extent it is justified. In this sense, it's a philosophical statement.

What principles lead us to believe that "Universes" is a safe assumption, whereas "¬Con(PA)" is not safe, regarding what we believe is "true" arithmetic?

There is an extensive philosophical literature discussing justifications for large cardinal axioms. You might start here. I'm not an expert, but I'm not aware of arguments which specifically argue that one should accept the arithmetic consequences of large cardinal axioms without arguing that one should accept the axioms outright. I would be interested to see such an argument. I once asked this question with related motivations.
I think the relevant thing to say is that I've never heard of similar sorts of justifications for $\neg Con(PA)$, and I don't expect I will.

(Next, repeat this question regarding the axiom of power set.)

Restrictions on powersets are studied in predicative mathematics. For some discussion of predicativism, one might start here or here. For some arguments for impredicativism, one might start here.

Is any theory that interprets PA "safe", as long as it is consistent with PA, and PA+Con(PA), and any such natural extension of these ideas?

By "safe", I take it that you continue to mean that the theory's arithmetic consequences are "true". There is a large body of literature discussing truth in mathematics. You seem to be particularly interested in the hierarchy obtained by passing from $T$ to $T + Con(T)$ iteratively. This hierarchy appears to be discussed from a philosophical perspective here. I think most logicians would probably agree that ascending this hierarchy adds relatively little consistency strength. And I'm not aware of serious attacks on the common-sense idea that if one believes $T$ is true, then one should also believe that $T + Con(T)$ is true.

Again, I'm not an expert, but it seems to me that most of the time when trying to justify stronger axioms from weaker assumptions, one appeals to some sort of reflection principle. This concept seems to be relevant to several of the questions you're asking.
You might also be interested in Reverse mathematics, where the "strength" of a mathematical theorem is calibrated by finding just how weak of a foundational theory can prove the theorem. One can interpret this as finding how "safe" a theorem is -- if you can prove it in a weaker theory, then it will continue to be true even if stronger theories are inconsistent, or simply "false".
A: Let us abbreviate ZFCU = ZFC + "Every set is contained in a Grothendieck universe."  Here is why ZFCU is a "safe" assumption.  Suppose it actually is consistent.  Then it cannot possibly prove any arithmetical statement that contradicts an arithmetical theorem of, for example, $n^{th}$-order PA for any $n$.  This is because it proves that $n^{th}$-order PA has an "internally standard" interpretation, and so everything that it proves about the naturals is true of the "standard" model (i.e. within the theory ZFCU), and thus agrees with everything "true" (true according to ZFCU).
If we further assume that ZFCU has an actually standard model, then this means that all its arithmetical theorems are actually true.  But this kind of begs the question, I think.  We would like to know about some kind of coherency between various theories at a syntactical level, that they don't prove contradictory arithmetical statements (or other statements about "standard objects").
The idea, I believe, is that if we can organize them linearly in a "standard interpretation hierarchy," then we are fine as long as we believe in the consistency of the strongest theory under consideration.  To be more precise, we look at theories $T$ which have a "natural numbers object" $\mathbb N^T$, which should at least satisfy PA.  If $S$ is another such theory, and $T$ proves that $S$ has a model $\frak A$ such that $\mathbb N^{\frak A} = \mathbb N^T$, then $T$ and $S$ cannot disagree about what their respective natural numbers object satisfies.
I don't at present have an all-encompassing definition of "standard model," but here are some examples to illustrate the idea.
Example: ZFC proves that for all $n<\omega$, $\mathbb N$ satisfies $n^{th}$-order PA.
Example: ZFCU proves that for each $n<\omega$, there is a standard model of $I_n = $ ZFC + "There are exactly $n$ inaccessible cardinals."  This should be the least rank $V_\alpha$ satisfying this theory, so that it is definable.  Although the $I_n$ are mutually inconsistent and inconsistent with ZFCU, whatever $I_n$ proves, ZFCU proves it holds in $I_n$'s standard model.  $I_n$ has a standard model of $I_m$ for $m<n$.  Iterating the operation of taking standard models "commutes" in some sense.
Non-example: ZFCU proves the consistency of $I_1$ + $\neg Con(I_2)$.  However, it proves that this does not hold in the standard model.
A: For what it's worth, the reason I made that comment was because when I gave talks expressing skepticism about the philosophical basis of ZFC, I would often get the reaction "but as long as it's consistent, what's the problem?"  Some version of this attitude is also quite prevalent in the philosophical literature on the subject.  I wanted to make the point that just being consistent isn't good enough: ZFC might well be consistent yet still prove, for example, that some Turing machine halts when in fact it does not.  If you're a skeptic about sets but not about numbers, this could be a concern.
As for the "safety" of assuming ZFC is arithmetically sound, it is clear that as long as ZFC is consistent, no weaker system could prove that it is unsound.  So for predicativists like me, who generally work in much weaker systems than ZFC, it seems unlikely that we could ever establish unsoundness.  In that sense I'd say it's pretty "safe" to assume it is sound.
However ... I regard the Peano axioms as expressing intuitively evident truths about the natural numbers, so I have no problem affirming Con(PA). Whereas I regard ZFC as what you get when you realize that full comprehension leads to inconsistencies, so you replace it with a hodgepodge of instances of comprehension which appears to be weak enough to block any paradoxes. (Edit. In case my meaning isn't clear: the axiom of pairing is an instance of full comprehension, the axiom of unions is an instance of full comprehension, the axiom of power sets is an instance of full comprehension ... Basically what you're doing is throwing in a bunch of unrelated instances of "for any property $P$ there is a set of all $x$ such that $P(x)$.") It seems to me quite ad hoc and unmotivated, so although we might (or might not) be confident that ZFC is consistent, there is little reason to expect it to be (or not to be) arithmetically sound.
(I should add that I regard the "iterative conception" which is supposed to justify ZFC as utterly unconvincing.  Sets are said to be built up in stages, and then it is parenthetically added that they are, of course, timeless abstract objects so they aren't really "built", nor do they really "appear" in "stages" --- it's all just a "metaphor". Thus "any conviction that the iterative conception may carry is made to depend on metaphorical details that are dismissed as inessential to it." (I. Jane))
A: This answer repeats some of the material in other answers but I think it is not entirely redundant.
As you seem to have realized, the assertion that a theory is consistent is a much weaker statement than the assertion that the theory does not prove false arithmetical statements (or is "arithmetically sound", to use the standard terminology). So the answer to your question about whether it could be the case that ZFC + universes is consistent but not arithmetically sound is yes, but for perhaps a trivial reason: ZFC + universes could be consistent and yet ZFC itself could fail to be arithmetically sound.  Probably what you really wanted to ask was, if ZFC is arithmetically sound, and ZFC + universes is consistent, does it follow that ZFC + universes is arithmetically sound?  The answer is no.  For example, it is conceivable that under these hypotheses, ZFC + universes could prove $\neg$Con(ZFC). (EDIT: This was a typo. I meant that ZFC+universes could prove $\neg$Con(ZFC+universes). Thanks to Noah Schweber for catching this—see the comments.)
Any feeling that universes are a "safe" assumption must come from a direct assessment of the universes axiom itself.  The reasons are similar to the reasons that we have for believing that ZFC is arithmetically sound.
The ability to remove Grothendieck universes from the proof of Fermat's Last Theorem cannot be deduced from some simple abstract fact like "Fermat's Last Theorem has low quantifier complexity."  No meta-theorem of this sort is known. Eliminability must come from careful examination of the nitty-gritty details of the proof.
Your question about whether Con(PA) is philosophical or mathematical needs clarification.  Offhand, it sounds like a false dichotomy to me, predicated on dubious assumptions about the meaning of the words "philosophical" and "mathematical." Can you define precisely what you mean by a "mathematical" assumption?  Are you implicitly assuming that "mathematical" assumptions are rock-solid while "philosophical" ones are not?  If so, that in itself is a dubious assumption IMO.
