Why should we care about "higher infinities" outside of set theory? Let's say you are a prospective mathematician with some addled ideas about cardinality.
If you assumed that the natural numbers were finite, you'd quickly vanish in a puff of logic. :)
If you thought that natural numbers and reals had the same cardinality - measure theory would almost surely break down, and your assumption would conflict with any number of "completeness theorems" in analysis (like the Baire Category Theorem for instance).
However, let's say you concluded that there were only three types of cardinality - finite, countably infinite, and uncountable.
Would this erroneous belief conflict with any major theorems in analysis, algebra or geometry ? Would any fields of math - outside set theory - be clearly incompatible with your assumption ?
PS: Apologies for the provocative title. Hope the question is clear.
 A: You might take a look at the following preprint of A.D.R. Mathias:  "Strong Statements of Analysis". This paper deals with the same concerns you seem to have regarding the applicability of higher infinites to ordinary mathematics.  In this paper, Mathias deals with the following four statements of analysis, all of them 'classical' (this may be a matter of opinion but Mathias makes an argument that this is so):
(A) Every uncountable co-analytic set ($\mathbf {\Pi}^{1}_1$ set) has a nonempty perfect set.
(B) Every analytic ($\mathbf {\Sigma^{1}_1}$ ) number-game is determined.
(C) Every $\mathbf {\Sigma^{1}_2}$ set is universally Baire.
(D) Every $\mathbf {\Sigma^{1}_2}$ number-game is determined 
Mathias shows (though not directly since this is a survey article but by pointing to references where the proofs of the statements can be found):
Statement A holds if and only if for each $\alpha$$\in$ $\mathscr N$, the true $\omega_1$ is a strongly inaccessible cardinal in $L[\alpha]$.
Statement B is equivalent to the assertion that every real has a sharp.
Statement C is equivalent to the assertion that every ordinal has a sharp.
Statement D is equivalent to the following assertion:  $\forall$$\alpha$:$\in$$\mathscr N$$\exists$ $\zeta$:$\lt$$\omega_1$$\exists$$S$: $\subseteq$$\zeta$ $\alpha$$\in$$L[S]$&$($${\zeta}$ $is$ $a$ $Woodin$  $cardinal$$)^{L[S]}$.
Some further quotes regarding statements A--D from his paper may be of some help to you also:
"This statement [Statement D--my comment] is more striking than the one equivalent to Statement A, as it is countable ordinals this time which are behaving as large cardinals in certain inner models, rather than just $\omega_1$.
So small cardinals might have large cardinals in certain inner models.  Note that in all four statements an equivalence is being proved, not just an equiconsistency; and in all except case C, one of the statements is a simple statement purely about real numbers.  Thus the above examples show the necessity of introducing large-cardinal properties into mathematics."     
A: In line with Joel's answer and the theme that stronger set theories permit finer analysis of higher infinities, an example from commutative algebra suggesting the desirability of distinguishing more than three broad classes of cardinals is (due in final form to Eda):
$Hom(\mathbb{Z}^\kappa / \mathbb{Z}^{<\omega}, \mathbb{Z}) \neq \lbrace 0 \rbrace$ if and only if there exists an $\omega_1$-complete non-principal ultrafilter on $\kappa$.
Whether the class $\lbrace \kappa : Hom(\mathbb{Z}^\kappa / \mathbb{Z}^{<\omega}, \mathbb{Z}) \neq \lbrace 0 \rbrace \rbrace$ is non-empty will depend on whether there are measurable cardinals.
An example from measure theory relates to Fubini's theorem, a central result of core mathematics. In 1990, Joseph Shipman proved (following results of Harvey Friedman on Tonelli-type theorems) that strong versions are provable once real-valued measurable cardinals are around:
J. Shipman, Cardinal conditions for strong Fubini theorems, TAMS 1990
A: $\newcommand\ZFC{\text{ZFC}}$Perhaps it would be useful to mention that set theorists have, of course,
studied numerous weaker set theories, including some extremely
weak theories, which do not give rise to higher cardinalities. One
may interpret your question as: to what extent do these weak set
theories serve as a foundation of mathematics?
To be sure, set theorists generally study these weak theories not
as foundational theories, but rather because they want to
undertake certain set-theoretic constructions in some much
stronger theory, but the objects appearing in the construction are
transitive sets satisfying these weaker theories, and so they need
to know, for example, whether those objects are themselves closed
under certain constructions. If those constructions can be
undertaken in the weak theory, then they are.
To give a few examples, the theory known as $\ZFC^-$, which is
basically $\ZFC$ without the power set axiom (but see my
recent paper, What is the theory ZFC without power set? for what this means exactly), is
widely used in set theory and has an enormous number of natural
models, including the universe $H_{\kappa^+}$, in which every set
has cardinality at most $\kappa$ and $P(\kappa)$ does not exist as
a set, but only as a class. For example, in the universe
$H_{\omega_1}$, the theory $\ZFC^-$ holds, and every set is
countable. This is a very rich universe in which to undertake
classical mathematics: you have all the reals individually, but
you cannot form them into a set; but you can still consider
(definable) functions on the reals and so on. You just cannot put
them all together into a set.
The theory known as Kripke-Platek set theory
$\text{KP}$ is another intensely studied theory, particularly for
those doing set theory with the constructible universe and
admissible set theory, and knowing what can be proved in
$\text{KP}$ and what cannot is very important in that area.
Even Zermelo set theory itself can be considered as a kind of example, since it does not prove
the existence of uncountable cardinals beyond the $\aleph_n$ for
$n<\omega$, because the rank-initial segment of the universe
$V_{\omega+\omega}$ is easily seen to be a model of Zermelo set
theory. So one could count this as a case of a weak theory that
does not prove a huge number of different infinities.
Perhaps this perspective on your question reveals that there is
really a continuum of such kind of answers. The really weak set
theories such as $\text{KP}$ and $\ZFC^-$ cannot prove even that
uncountable cardinals exist, but then slightly stronger theories,
which become true in $H_{\omega_2}$ or $H_{\omega_3}$, can prove a
few more uncountable cardinals. Zermelo's theory provides more,
but still only countably many uncountable cardinals. The $\ZFC$
theory of course then explodes with an enormous number of
different uncountable cardinals.
But let me say that this process continues strictly past $\ZFC$, for large cardinal set theorists look upon $\ZFC$
itself as a weak theory, in precisely this sense, because it
cannot prove the existence of measurable or supercompact
cardinals (and many others), for example, and so one must continue up the large cardinal
hierarchy in order to get the kinds of infinities that we like.
Set theorists consider theories all along the large cardinal
hierarchy, with the stronger theories giving us more and stronger
axioms of the higher infinite.
At every step of this entire hierarchy, starting from the very
weak theories I mentioned and continuing into the large cardinal
hierarchy, there are fundamental set-theoretic assertions that are
provable by the stronger theory but not provable by the weaker
theory.
Meanwhile, despite the fact that some every-day mathematical
objects have distinct uncountable cardinalities (and so the weak
set theories cannot prove they exist), nevertheless it is quite
surprising how close an approximation one can get just in
second-order number theory, where in a sense every object is
countable. The work of reverse mathematics generally takes place in the
context of second-order number theory, and seeks to find exactly
the theory that is necessary in order to prove each of the
classical theorems of mathematics. (Thus, they try to prove the
axioms from the theorem, rather than the other way.) They have
numerous examples of which classical theorems you can prove and
exactly what theory (provably so!) you need to do it.
A: 
Would this erroneous belief conflict with any major theorems in analysis, algebra or geometry ? Would any fields of math - outside set theory - be clearly incompatible with your assumption ?

There are natural theorems in analysis which you can no longer even state under your assumptions. There are natural theorems in analysis and algebra which you still can state, but no longer prove. For geometry, it depends a bit on what you still consider as belonging to geometry.

However, a real misunderstanding is that you assume the purpose of set theory would be to describe all collections (and urelements) that actually exists or could potentially exists. The entry on Alternative Axiomatic Set Theories by Randall Holmes in the Stanford Encyclopedia of Philosophy might help to get a better understanding of the role of set theories in the foundations of mathematics from a still quite mathematical point of view.
A set theory provides an ordered universe in which one can reason mathematically. Limiting the size of the universe somehow is often required to ensure some order, but size is not the only factor. A philosophical text like
On What There Is by Willard Van Orman Quine might get you some feeling why considering every potentially existing collection (or urelement) would be a breeding ground for disorderly elements:

Wyman's overpopulated universe is in many ways unlovely. It offends the aesthetic sense of us who have a taste for desert landscapes, but this is not the worst of it. Wyman's slum of possibles is a breeding ground for disorderly elements. Take, for instance, the possible fat man in that doorway; and, again, the possible bald man in that doorway. Are they the same possible man, or two possible men? How do we decide? How many possible men are there in that doorway? Are there more possible thin ones than fat ones? How many of them are alike? Or would their being alike make them one? Are no two possible things alike? Is this the same as saying that it is impossible for two things to be alike? Or, finally, is the concept of identity simply inapplicable to unactualized possibles? But what sense can be found in talking of entities which cannot meaningfully be said to be identical with themselves and distinct from one another? These elements are well-nigh incorrigible. By a Fregean therapy of individual concepts, some effort might be made at rehabilitation; but I feel we'd do better simply to clear Wyman's slum and be done with it.

A: "However, let's say you concluded that there were only three types of cardinality--finite, countably infinite, and uncountable."
So let's consider the countable ordinals, and certain combinatorial principles regarding them.  These come from Harvey Friedman (in this case from his preliminary draft (1995) "Combinatorial Set Theoretic Principles of Great Logical Strength" found on his Homepage).
First, some preliminary definitions from his report:
Definition.  Let $j$:$\beta$$\rightarrow$$\beta$, where $\beta$ is an ordinal.  Let $R$$\subseteq$ $\alpha$ x $\alpha$, where $\beta$$\le$$\alpha$.  We define $j$[$R$]={$j$(c),$j$(d): $R$(c,d)}.  we say that $j$ is a nonidentity function if and only if $j$ is not the identity function on $\beta.$
Definition.  Let $F$:$\alpha$ x $\alpha$$\rightarrow$$\alpha$.  For $\beta$$\lt$$\alpha$, we write $F$$\beta$ for the restriction of the cross section of $F$ to $\beta$; i.e., $F$$\beta$(c)=d if and only if c$\lt$b and $F$(b,c)=d.  [Note:  should b=$\beta$ in this definition?--my comment]
$LCA_1$.  There is a nontrivial elementary embedding of $V$$\rightarrow$$M$ such that $V$($\alpha$)$\subseteq$$M$, where $\alpha$ is the first fixed point above the critical point.
$LCA_2$.  There is a nontrivial elementary embedding of a rank into itself.
Here $LCA$ stands for "large cardinal axiom". 
The combinatorial principles relevant to countable ordinals are $P_4$($\alpha$), $A_4$, and $A_5$.  Here are their definitions:
$P_4$($\alpha$).  There exists $F$: $\alpha$ x $\alpha$$\rightarrow$$\alpha$ such that for every $R$$\subseteq$$\alpha$ x $\alpha$ that is constructible from $F$, there is a nonidentity $F$$\beta$:$\beta$$\rightarrow$$\beta$, $\beta$$\lt$$\alpha$, such that $F$$\beta$[$R$]$\subseteq$$R$.
$A_4$.  $P_4$($\alpha$) holds for some countable $\alpha$.
$A_5$.  There is a countable function of the form $F$: $\alpha$ x $\alpha$$\rightarrow$$\alpha$ such that for every unbounded $R$$\subseteq$$\alpha$ x $\alpha$ that is first order definable in ($\alpha$, $\lt$, $F$), some $F$$\beta$ is a nontrivial injection (embedding) of a proper initial segment of $R$.
Now for the relevant theorems, which are stated without proof in the report (you might email Prof. Friedman and ask for them...):
Theorem 5.  $ZFC$ $+$ $LCA_1$ implies $A_4$.    $ZF$ $+$ $A_4$ implies the existence of a transitive class containing all ordinals which satisfies $ZFC$ $+$ $LCA_2$.
Theorem 6.  $ZFC$ $+$ $LCA_1$ implies $A_5$.    $ZF$ $+$ $A_5$ implies the existence of a transitive model of $ZFC$ $+$ $LCA_2$ 
Another preliminary report of Friedman's you might also find interesting is "Extremely Large Cardinals in the Rationals".   It has results somewhat similar to these.  
