So far the discusion mostly focused on a geometric explanation, I'd like to mention the algebraic one as well:

One way to formulate it involves the delooping machinery: up to delooping, the $\mathbb{S}^n$ corresponds to the free group like $E_n$-algebra on one generator.

(Small recall: the usual delooping machinery say that the looping/delooping construction induce an equivalence between pointed spaces $X$ such that $\pi_k X =0$ for all $k<n$ and group like $E_n$-algberas. Through that correspondence $\mathbb{S}^n$ corresponds to the free group like $E_n$-algebra on one generators, as the looping/delooping adjunction justs shift the $\pi_n$ you can describe the homotopy group of sphere as shifted homotopy group of these free group like $E_n$-algebra.)


Now when you construct the free $E_1$-algebra, there not much you can do: you only have one way to multiply elements and the free $E_1$-algebra on one generator is just the monoid $\mathbb{N}$ (and $\mathbb{Z}$ for the group like one).

But for $E_n$ you get $n$ "compatible" (in a homotopy theoretic sense) way to multiply the elements and all the higher elements in the homotopy group comes from the interaction (the coherence law, that are given by homotopies) between these multiplications, for example the free $E_2$-algebra has all the braid groups appearing as its various $\pi_1$ due to that, and the free group like $E_2$-algebra become too complicated to described (well... it is $\Omega^2 \mathbb{S}^2$ ).

So the difference is that for $n=1$ no such interaction is happening because to get interaction you need at least two compatible multiplication.


Alternatively to the delooping machinery, one can (somehow equivalently) think of spaces as $\infty$-groupoids and as the $n$-sphere as the $\infty$-groupoid freely generated by a cell in dimension $n$. The discussion is pretty much the same except that now the $n$ "compatible" multiplication are simply the compositions in direction $k$ for $k$ from $0$ to $n-1$.


Edit: Here is how you get a non trivial element of $\pi_3(\mathbb{S}^2)$, in the second perspective. I'm using an unspecified model of weak $\infty$-groupoid, and applying freely the operation of strict $\infty$-categories to give a feel of how it works, this is not mean't to be formal (but it is formalizable in any algebraic model of weak $\infty$-groupoid, or in Hott )

Given a two cells $u$ and $v$ whose source and target is a (weak) identity, the usual Eckman Hilton argument (so the typical example of interaction between $\#_0$ and $\#_1$ as I mentioned above) gives an isomorphism $\theta_{u,v} : u \#_0 v \simeq v \#_0 u$.

If $e$ is the generating 2-cell of the 2sphere then this gives an isomorphisms $\theta_{e,e}: e \#_0 e \simeq e \#_0 e $ 
taking $e^*$ a $0$ inverse of $e$, one has that $e^* \#_0 \theta_{e,e} \# e^*$ is a 3-cell whose source and target are (up to the coherence isomorphism expressing that $e$ and $e^*$ are inverse) identities, so it gives an element of $\pi_3(\mathbb{S}^2)$, which is non-zero by a universality argument. I'm convince it is a generator (so either the Hopf fibration or its opposite depending which way you have rotated the 'Eckman-Hilton clock') but I don't know how to prove it using only this type of tools.