So far the discusion mostly focused on a geometric explanation, I'd like to mention the algebraic one as well:

One way to formulate it involves the delooping machinery: up to delooping, the $n$-sphere corresponds to the free group like $E_n$-algebra on one generators.

(Small recall: the usual delooping machinery say that the looping/delooping construction induces an equivalence between pointed spaces $X$ such that $\pi_k X =0$ for all $k<n$ and group like $E_n$-algberas. Through that correspondences the n-sphere corresponds to the free group like $E_n$-algebra on one generators, as the looping/delooping adjunction justs shift the $\pi_n$ you can describe the homotopy group of sphere as shifted homotopy group of these free group like $E_n$-algebra.)


Now when you construct the free $E_1$-algebra, there not much you can do: you only have one way to multiply elements and the free $E_1$-algebra on one generators is just $\mathbb{N}$ (and $\mathbb{Z}$ for the group like one)

But for $E_n$ you get $n$ "compatible" (in a homotopy theoretic sense) way to multiply the elements and all the higher elements in the homotopy group comes from the interaction (the coherence law, that are given by homotopies) between these multiplications, for example the free $E_2$-algebra has braid groups appearing as its $\pi_1$ due to that, and the free group like $E_2$-algebra become too complicated to described (well... it is essentially $\Omega^2 \mathbb{S}^2$ ).

So the difference is that for $n=1$ no such interaction is happening because to get interaction you need at least two compatible multiplication.


Alternatively to the delooping machinery, one can (somehow equivalently) think of spaces as $\infty$-groupoids and as the $n$-sphere as the $\infty$-groupoid freely generated by a cell in dimension $n$. The discussion is pretty much the same except that now the $n$ "compatible" multiplication are simply the compositions in direction $k$ for $k$ from $0$ to $n-1$.