An elementary answer to the first part of your question: Finite sets are more fundamental than their cardinalities.
Consider the category of finite sets and bijective functions. Its geometric realization (= nerve, or classifying space) has the homotopy type of $\coprod_{n\ge0} B\Sigma_n$. Forgetting the choices of bijective functions, and only remembering their existence, gives a map to the set $\mathbb{N}_0 = \{n \ge 0\}$ of non-negative integers. This way the symmetries of a finite set are disregarded: you can either fix or transpose the two elements in $\{a, b\}$, whereas the number $2$ does not intrinsically come with such structure.
Soon you want to add and multiply finite sets, using disjoint union and cartesian product, and these operations on $\coprod_{n\ge0} B\Sigma_n$ induce the usual sum and product in $\mathbb{N}_0$. We now have a map of semirings.
Shortly thereafter you want to solve equations, and need to subtract. For this, you need to ring complete the addition in $\coprod_{n\ge0} B\Sigma_n$, while preserving the multiplication (see Remark A), and the resulting ring space has the homotopy type of $\text{colim}_k \, \Omega^k S^k = \Omega^\infty \mathbb{S}$, with the ring space structure coming from the ring spectrum $\mathbb{S}$, the sphere spectrum. This now maps to the ring completion of $\mathbb{N}_0$, namely the integers $\mathbb{Z}$.
The first thing that is lost under $\mathbb{S} \to \mathbb{Z}$ is the image of the transposition of $a$ and $b$, i.e., the Mobius strip/line bundle over the circle, which corresponds to the loop in $\Omega^2 S^2$ given by the Hopf fibration $\eta \colon S^3 \to S^2$.
Remark A: One way to resolve the caveat raised in
Thomason, R. W.
Beware the phony multiplication on Quillen's A−1A.
Proc. Amer. Math. Soc. 80 (1980), no. 4, 569–573.
is given in
Baas, Nils A.; Dundas, Bjørn Ian; Richter, Birgit; Rognes, John
Ring completion of rig categories.
J. Reine Angew. Math. 674 (2013), 43–80.