Let $G$ be a group, and consider the matrix elements of finite dimensional irreducible unitary representations of $G$ over $\mathbb{C}$ as functions $f:G\to \mathbb{C}$. If $G$ is finite, any two such functions are orthogonal under the usual inner product $$ \langle f_1,f_2\rangle=\frac{1}{|G|}\sum_{g\in G} f_1(g)\; \overline{f_2(g)} $$ and the inner product of a matrix element with itself equals the inverse of the dimension of the representation. A corollary is the orthonormality of characters.

Is there a generalisation of this inner product to (certain classes of) infinite discrete groups that reproduces these results?

I have in mind finitely generated groups, for which the sums certainly won't converge except in very special cases. One guess here is to pick finite subsets $G_1\subseteq G_2\subseteq \cdots$ such that $\bigcup_k G_k=G$ (for example, arranging by minimal length of generating word), and take $\lim_{i\to\infty}\frac{1}{|G_k|}\sum_{g\in G_k}$. The hope is that this would converge to the desired results for a class of `reasonable' choices of $G_k$. It works for the simplest example of $\mathbb{Z}$, choosing the $G_k=\{n_k,n_k+1,\ldots,n_{k}+k\}$ for some appropriate $n_k$: the irreducible unitary representations are one-dimensional, labelled by $e^{i\theta}\in S_1$, and the result follows straightforwardly.

compact groups; e.g. for compact groups you still get all irreps being fin-dim, you have Schur orthogonality for matrix coefficients of irreps, and so on. Other than Z, which other groups did you have in mind? $\endgroup$ – Yemon Choi May 24 '16 at 23:37