First, I think it is better to restrict the term "Fourier Analysis" to refer to the process of expanding functions on a locally compact ABELIAN group $G$ as a "sum'' of the characters of the group. (I'll come back to that in a moment.) The generalization, when the group $G$ is not assumed to be abelian, should probably be better referred to as Harmonic Analysis". Regarding the latter, if the group $G$ is compact, then the Peter-Weyl Theorem gives an elegant and simple generalization to the theory of Fourier Series on the circle group---it shows how to write any $L^2$ function on $G$ as an series of (orthogonal) matrix elements of irreducible unitary representations of $G$. When $G$ is neither abelian nor compact, the theory becomes MUCH more complicated and sophisticated. BTW, note that when $G$ is abelian, then as you pointed out, the irreducble unitary representaions of $G$ are one-dimensional, so there is no difference between a matrix element and a character in this case and we are generalizing Fourier series on the circle group.
OK, lets now restrict to the Fourier" case, where $G$ is locally compact and abelian. Note that an irreducible unitary character of $G$ is now just a group homomorphism of $G$ into the circle group $S= S^1$ (considered as the complex numbers of modulus one under multiplication). Since $G$ is abelian, the set $\hat G = Hom(G,S)$ is an abelian group, the character (or Pontrjagin dual) group of $G$, under pointwise multiplication. It is easy to see that $\hat G$ is locally compact (in the compact open topology) What Fourier analysis becomes in this case is a method for expressing an arbitrary element of $L^2(G)$ as an integral of the form $f(g) \sim \int \hat f(\chi)\chi(g) dg$d\chi$, where$\hat f$, the Fourier transform of$f$is defined dually by$\hat f(\chi) = \int f(g) \chi(g) dg$(and the Haar measures on$G$and$\hat G$are suitably normalized). Note that if we take for$G$the real line$R$then this reduces to the classical Fourier transform. It is easy to show that the integral defining the Fourier transform$\hat f(\chi)$is convergent when$f$is in$L^1 \cap L^2$and that then$||\hat f||_2 = ||f||_2^2$, and since$L^1 \cap L^2$is dense in$L^2$it follows that the Fourier transform extends uniquely to a unitary map of$L^2(G)$onto$L^2(\hat G)$. Now lets restrict further to the compact case, where characters, being continuous, are bounded and so integrable. As one can prove in a couple of lines (using the invariance of Haar measure), if$\chi$is any character of$G$then$\int \chi(g)\, dg = 0$unless$\chi$is the identity character in which case the integral is one (using normalized Haar measure on$G$). Since the complex conjugate of a character is its inverse in$\hat G$, it now follows trivially that the elements of$\hat G$are orthonormal. In fact they form an orthonormal basis for$L^2(G)$, and the Fourier transform of the preceding paragraph becomes a formula for expanding any element of$L^2(G)$as the sum of an infinite series in the characters of$G$, a direct generalization of the theory of Fourier series (the case when$G = S$). A good place to see all the details is Lynn Loomis' "Absract Harmonic Analysis". 2 added 101 characters in body; added 6 characters in body First, I think it is better to restrict the term "Fourier Analysis" to refer to the process of expanding functions on a locally compact ABELIAN group$G$as a "sum'' of the characters of the group. (I'll come back to that in a moment.) The generalization, when the group$G$is not assumed to be abelian, should probably be better referred to as Harmonic Analysis". Regarding the latter, if the group$G$is compact, then the Peter-Weyl Theorem gives an elegant and simple generalization to the theory of Fourier Series on the circle group---it shows how to write any$L^2$function on$G$as an series of (orthogonal) matrix elements of irreducible unitary representations of$G$. Note When$G$is neither abelian nor compact, the theory becomes MUCH more complicated and sophisticated. BTW, note that when$G$is abelian, then as you pointed out, the irreducble unitary representaions of$G$are one-dimensional, so there is no difference between a matrix element and a character in this case and we are generalizing Fourier series on the circle group. OK, lets now restrict to the Fourier" case, where$G$is locally compact and abelian. Note that an irreducible unitary character of$G$is now just a group homomorphism of$G$into the circle group$S= S^1$(considered as the complex numbers of modulus one under multiplication). Since$G$is abelian, the set$\hat G = Hom(G,S)$is an abelian group, the character (or Pontrjagin dual) group of$G$, under pointwise multiplication. It is easy to see that$\hat G$is locally compact (in the compact open topology) What Fourier analysis becomes in this case is a method for expressing an arbitrary element of$L^2(G)$as an integral of the form$f(g) \sim \int \hat f(\chi)\chi(g) dg$, where$\hat f$, the Fourier transform of$f$is defined dually by$\hat f(\chi) = \int f(g) \chi(g) dg$(and the Haar measures on$G$and$\hat G$are suitably normalized). Note that if we take for$G$the real line$R$then this reduces to the classical Fourier transform. It is easy to show that the integral defining the Fourier transform$\hat f(\chi)$is convergent when$f$is in$L^1 \cap L^2$and that then$||\hat f||_2 = ||f||_2^2$, and since$L^1 \cap L^2$is dense in$L^2$it follows that the Fourier transform extends uniquely to a unitary map of$L^2(G)$onto$L^2(\hat G)$. Now lets restrict further to the compact case, where characters being continuous are bounded and so integrable. As one can prove in a couple of lines (using the invariance of Haar measure), if$\chi$is any character of$G$then$\int \chi(g)\, dg = 0$unless$\chi$is the identity character in which case the integral is one (using normalized Haar measure on$G$). Since the complex conjugate of a character is its inverse in$\hat G$, it now follows trivially that the elements of$\hat G$are orthonormal. In fact they form an orthonormal basis for$L^2(G)$, and the Fourier transform of the preceding paragraph becomes a formula for expanding any element of$L^2(G)$as the sum of an infinite series in the characters of$G$, a direct generalization of the theory of Fourier series (the case when$G = S$). A good place to see all the details is Lynn Loomis' "Absract Harmonic Analysis". 1 First, I think it is better to restrict the term "Fourier Analysis" to refer to the process of expanding functions on a locally compact ABELIAN group$G$as a "sum'' of the characters of the group. (I'll come back to that in a moment.) The generalization, when the group$G$is not assumed to be abelian, should probably be better referred to as Harmonic Analysis". Regarding the latter, if the group$G$is compact, then the Peter-Weyl Theorem gives an elegant and simple generalization to the theory of Fourier Series on the circle group---it shows how to write any$L^2$function on$G$as an series of (orthogonal) matrix elements of irreducible unitary representations of$G$. Note that when$G$is abelian, then as you pointed out, the irreducble unitary representaions of$G$are one-dimensional, so there is no difference between a matrix element and a character in this case and we are generalizing Fourier series on the circle group. OK, lets now restrict to the Fourier" case, where$G$is locally compact and abelian. Note that an irreducible unitary character of$G$is now just a group homomorphism of$G$into the circle group$S= S^1$(considered as the complex numbers of modulus one under multiplication). Since$G$is abelian, the set$\hat G = Hom(G,S)$is an abelian group, the character (or Pontrjagin dual) group of$G$, under pointwise multiplication. It is easy to see that$\hat G$is locally compact (in the compact open topology) What Fourier analysis becomes in this case is a method for expressing an arbitrary element of$L^2(G)$as an integral of the form$f(g) \sim \int \hat f(\chi)\chi(g) dg$, where$\hat f$, the Fourier transform of$f$is defined dually by$\hat f(\chi) = \int f(g) \chi(g) dg$(and the Haar measures on$G$and$\hat G$are suitably normalized). Note that if we take for$G$the real line$R$then this reduces to the classical Fourier transform. It is easy to show that the integral defining the Fourier transform$\hat f(\chi)$is convergent when$f$is in$L^1 \cap L^2$and that then$||\hat f||_2 = ||f||_2^2$, and since$L^1 \cap L^2$is dense in$L^2$it follows that the Fourier transform extends uniquely to a unitary map of$L^2(G)$onto$L^2(\hat G)$. Now lets restrict further to the compact case, where characters being continuous are bounded and so integrable. As one can prove in a couple of lines (using the invariance of Haar measure), if$\chi$is any character of$G$then$\int \chi(g)\, dg = 0$unless$\chi$is the identity character in which case the integral is one (using normalized Haar measure on$G$). Since the complex conjugate of a character is its inverse in$\hat G$, it now follows trivially that the elements of$\hat G$are orthonormal. In fact they form an orthonormal basis for$L^2(G)$, and the Fourier transform of the preceding paragraph becomes a formula for expanding any element of$L^2(G)$as the sum of an infinite series in the characters of$G$, a direct generalization of the theory of Fourier series (the case when$G = S\$).