It isn't an algebra: the tensor product of irreducibles does not always decompose as a finite sum of irreducibles, although in some of those cases it may decompose as a direct integral of irreducibles.

For instance, take $G=H_3({\bf R})$, the group of upper-triangular matrices with real entries and with $1$ on the diagonal. This has very nice representation theory from the point of view of harmonic analysis (it is not only Type I, but it is what Kaplansky called CCR, that is $\pi(L^1(G))\subseteq {\mathcal K}(H_\pi)$ for every continuous irreducible unitary representation $\pi: G \to {\mathcal U}(H_\pi)$). There is a continuous quotient homomorphism $q:G \to G/Z(G) \cong {\bf R}^2$.

Now for $t\neq 0$ there is an infinite irreducible unitary representation $\pi_t: G \to {\mathcal U}(L^2({\bf R}))$ (in some realizations called the *Schrödinger representation*). It is true that if $s$, $t$ and $s+t$ are all non-zero, and $f$ is a coefficient function of $\pi_s$ and $g$ is a coefficient function of $\pi_t$, then $fg$ is the norm-limit inside $B(G)$ of sums of coefficient functions of $\pi_{s+t}$. I don't know if $fg$ is actually a coefficient function itself.

However, the representation $\pi_t\otimes \pi_{-t}$ is equivalent to the representation $\lambda_{\bf R{^2}} \circ q: G \to {\mathcal U}(L^2({\bf R}^2))$ which is not (quasi-)equivalent to a direct sum of irreducibles. Consequently there should exist $f$ a coefficient function of $\pi_t$ and $g$ a coefficient function of $\pi_{-t}$ such that $fg$ does not belong to the closure of ${\rm Trig}(G)$ inside $B(G)$.

Similar example should exist for e.g. ${\rm SL}(2,{\bf R})$ but I don't remember the ``fusion rules'' for its irreducible representations off the top of my head.

On the positive side: if $G$ is a Moore group then the completion of ${\rm Trig}(G)$ inside $B(G)$ is indeed a subalgebra of $B(G)$. This may be folklore: I learned of it from the paper http://arxiv.org/abs/1208.1519 which has more information on "negative examples".

3more comments