It is not hard to find the spectral theory of a single unitary operator $U$. This is the spectral theory for a $\mathbb{Z}$action because we consider $U^n$ for $n\in\mathbb{Z}$. This is clear with eigenvalues because given an eigenvalue $\lambda$ with eigenvector $v$, we have $U^nv=\lambda^nv$. Given a Banach space $B$ and a group $G$ and a homomorphism $\pi:G\to L(B,B)$ where $\pi(g)$ unitary $\forall g \in G$. We can define an eigenvalue $\lambda$ of this $G$action to be: $\exists v\ne0$ such that $\pi(g)v=\lambda(g)v$ where we see $\lambda$ is a group homomorphism from $G$ to $\mathbb{C}$. So we know eigenvalues, but what about generalizations of other parts of the spectrum? Is there a developed theory of this somewhere?

As posed, the question turns into a question about repns about abelian groups $G$, since the $\lambda$ necessarily factors through the derived group $G/[G,G]$. This commutativity assures that eigenspaces (such as there are) are respected, for example. We might hope that the operators attached to $G$ are all normal, so that the usual spectral "calculus" applies, ... in which case there is a simultaneous "diagonalization" of all the operators coming from $G$. (Edit: Gelfand's theory of commutative $C^*$algebras is a nice package for this.) To lift the effectivelyabelian constraint, one must moreorless sacrifice the notion of "eigenspace/eigenvalue" and retool... to ask about irreducible subrepresentations. There is some cost to this, but it's certainly worth it in the long term. Many natural situations require this, e.g., the action of the rotation group on $L^2$ functions on the 2sphere (in Euclidean 3space): there are no nontrivial group homs of the rotation group to $\mathbb C^\times$, but $L^2$ is the direct sum of the irreducible subspaces consisting of homogeneous harmonic polynomials (restricted to the sphere) of various degrees. In the latter example, as in many others, but not universally, the irreducibles are also characterizable as eigenspaces for a single operator. In the case of spheres, this is the (nonEuclidean) Laplacian, which (by luck) distinguishes the irreducibles occuring in $L^2(S^2)$ by its eigenvalues. 

