The Barratt-Priddy-Quillen(-Segal) theorem says that the following spaces are homotopy equivalent in an (essentially) canonical way:
$\Omega^\infty S^\infty:=\varinjlim~ \Omega^nS^n$
$\mathbb{Z}\times ({B\Sigma_\infty})_+$, where $\Sigma_\infty$ is the group of automorphisms of a countable set which have finite support, and $+$ is the Quillen plus-construction.The group completion of $B\left(\bigsqcup_n \Sigma_n\right)$, where $\Sigma_n$ is the symmetric group on $n$ letters, and $B(\sqcup_n \Sigma_n)$ is given the structure of a topological monoid via the block addition map $\Sigma_n\times \Sigma_m\to \Sigma_{n+m}$.
$\Omega|S^\bullet.\operatorname{FinSet}|$, where $S^\bullet$ is the Waldhausen $S$-construction, and $\operatorname{FinSet}$ is the category of pointed finite sets, given the structure of a Waldhausen category by declaring cofibrations to be injections and weak equivalences to be isomorphisms. I don't want to define this since it's complicated (for a reference, see Chapter IV of Weibel's K Book), but it should be thought as a homotopical version of the Grothendieck ring of finite sets, where addition is given by disjoint union and multiplication is given by the cartesian product. Clark Barwick's answer here makes this more precise.
Now, the homotopy groups of the first space are manifestly the stable homotopy groups of spheres; on the other hand, the last two spaces clearly encode some information about the combinatorics of finite sets. So my question is:
Is there a concrete combinatorial interpretation of the higher stable homotopy groups of spheres in terms of the combinatorics of finite sets or symmetric groups?
For example it is easy to see via (3) or (4) that $\pi_{0+k}(S^k)=\mathbb{Z}$ corresponds to the Grothendieck ring of finite sets. Similarly, (2), or with some theory (4), make it clear that $\pi_{1+k}(S^k)=\mathbb{Z}/2\mathbb{Z}$ corresponds to the abelianization of $\Sigma_n$ (via the sign homomorphism). I am interested in concrete interpretations of the higher stable homotopy groups in this style.
A good answer would be, for example, a direct combinatorial interpretation of $\pi_{2+k}(S^k)=\mathbb{Z}/2\mathbb{Z}$ and $\pi_{3+k}(S^k)=\mathbb{Z}/24\mathbb{Z}$; a not-so-good answer would be a statement like "the sphere spectrum is a categorification of the integers," which is not the sort of concrete thing I'm looking for.
EDIT: So with the exception of Jacob Lurie's comment on $\pi_{2+k}(S^k)$ below (interpreting it as the Schur multiplier $H_2(\Sigma_\infty, \mathbb{Z})$ of $\Sigma_\infty$), it seems like it might be too much to hope for any reasonably complete combinatorial interpretation of the stable homotopy groups. So I'd settle for something like the following: namely, a sequence of groups $G_n$ defined in some combinatorial way, and maps $f_n: G_n\to \pi_{n+k}(S^k)$ or $g_n: \pi_{n+k}(S^k)\to G_n$ such that
$f_n$ or $g_n$ are nontrivial for infinitely many $n$,
The maps are related in some way to the constructions 2-4 above, and
The $G_n$ are combinatorially interesting.
One such example is $G_n:=H_n(\Sigma_\infty, \mathbb{Z})$ with $g_n$ the Hurewicz map (whence the interpretation of $\pi_{2+k}(S^k)$). But even in this case, the combinatorial meaning is sort of mysterious (to me at least) for large $n$.

