First, I think it is better to restrict the term "Fourier Analysis" to refer to the process of expanding functions on a locally compact ABELIAN group $G$ as a "sum'' of the characters of the group. (I'll come back to that in a moment.) The generalization, when the group $G$ is not assumed to be abelian, should probably be better referred to as ``Harmonic Analysis". Regarding the latter, if the group $G$ is compact, then the Peter-Weyl Theorem gives an elegant and simple generalization to the theory of Fourier Series on the circle group---it shows how to write any $L^2$ function on $G$ as an series of (orthogonal) matrix elements of irreducible unitary representations of $G$. When $G$ is neither abelian nor compact, the theory becomes MUCH more complicated and sophisticated. BTW, note that when $G$ is abelian, then as you pointed out, the irreducble unitary representaions of $G$ are one-dimensional, so there is no difference between a matrix element and a character in this case and we are generalizing Fourier series on the circle group.
OK, lets now restrict to the ``Fourier" case, where $G$ is locally compact and abelian. Note that an irreducible unitary character of $G$ is now just a group homomorphism of $G$ into the circle group $S= S^1$ (considered as the complex numbers of modulus one under multiplication). Since $G$ is abelian, the set $\hat G = Hom(G,S)$ is an abelian group, the character (or Pontrjagin dual) group of $G$, under pointwise multiplication. It is easy to see that $\hat G$ is locally compact (in the compact open topology) What Fourier analysis becomes in this case is a method for expressing an arbitrary element of $L^2(G)$ as an integral of the form $f(g) \sim \int \hat f(\chi)\chi(g) dg$d\chi$, where $\hat f$, the Fourier transform of $f$ is defined dually by $\hat f(\chi) = \int f(g) \chi(g) dg$ (and the Haar measures on $G$ and $\hat G$ are suitably normalized). Note that if we take for $G$ the real line $R$ then this reduces to the classical Fourier transform. It is easy to show that the integral defining the Fourier transform $\hat f(\chi)$ is convergent when $f$ is in $L^1 \cap L^2$ and that then $||\hat f||_2 = ||f||_2^2$, and since $L^1 \cap L^2$ is dense in $L^2$ it follows that the Fourier transform extends uniquely to a unitary map of $L^2(G)$ onto $L^2(\hat G)$.
Now lets restrict further to the compact case, where characters, being continuous, are bounded and so integrable. As one can prove in a couple of lines (using the invariance of Haar measure), if $\chi$ is any character of $G$ then $\int \chi(g)\, dg = 0$ unless $\chi$ is the identity character in which case the integral is one (using normalized Haar measure on $G$). Since the complex conjugate of a character is its inverse in $\hat G$, it now follows trivially that the elements of $\hat G$ are orthonormal. In fact they form an orthonormal basis for $L^2(G)$, and the Fourier transform of the preceding paragraph becomes a formula for expanding any element of $L^2(G)$ as the sum of an infinite series in the characters of $G$, a direct generalization of the theory of Fourier series (the case when $G = S$).
A good place to see all the details is Lynn Loomis' "Absract Harmonic Analysis".