I have been struggling with very similar issues and was almost ready to ask here, but then found this question, so I'll instead post my ramblings as an "answer" to avoid duplication of questions (of course, it doesn't answer anything).

The official story seems to go something like this (for example in Weyl's classic): Look at something like $H=-\Delta + V(|x|)$ on $L^2(\mathbb R)$. Then the natural action of rotations $R\in SO(3)$ on $L^2$ by the unitary maps $(R\cdot f)(x) = f(R^{-1}x)$ is a symmetry of the system; this feels obvious, especially if I don't specify what I mean by symmetry. "Therefore" the irreducible representations of $SO(3)$ must be important.

I would really love to have this "therefore" explained in mathematical style, but after some reading around, my current impression is that there is nothing of the sort available. Rather, it seems to be a vaticinium ex eventu: For example, the spherical harmonics play a key role when analyzing $H$ above, and these do give us finite-dimensional representations. The representation on $\mathbb C^2$ (and for some reason we have now also found it convenient to slightly change $SO(3)$ to $SU(2)$) shows up in the description of spin. "Therefore," we conclude that representations must be relevant in general.

Since this answer is written from a position of ignorance, comments are very welcome.

I have been struggling with very similar issues and was almost ready to ask here, but then found this question, so I'll instead post my ramblings as an "answer" to avoid duplication of questions (of course, it doesn't answer anything).

The official story seems to go something like this (for example in Weyl's classic): Look at something like $H=-\Delta + V(|x|)$ on $L^2(\mathbb R^3)$. Then the natural action of rotations $R\in SO(3)$ on $L^2$ by the unitary maps $(R\cdot f)(x) = f(R^{-1}x)$ is a symmetry of the system; this feels obvious, especially if I don't specify what I mean by symmetry. "Therefore" the irreducible representations of $SO(3)$ must be important.

I would really love to have this "therefore" explained in mathematical style, but after some reading around, my current impression is that there is nothing of the sort available. Rather, it seems to be a vaticinium ex eventu: For example, the spherical harmonics play a key role when analyzing $H$ above, and these do give us finite-dimensional representations. The representation on $\mathbb C^2$ (and for some reason we have now also found it convenient to slightly change $SO(3)$ to $SU(2)$) shows up in the description of spin. "Therefore," we conclude that representations must be relevant in general.

Since this answer is written from a position of ignorance, comments are very welcome.