Trace Class Functions on locally compact groups

Question

Let $G$ be a locally compact subgroup, $\mu$ a Haar-measure. For $f \in L^1(G)$, and for $\pi$ a unitary, topology irreducible, representation of $G$ on an Hilbert space $H_\pi$, it is customary to define a continuous operator $\pi(f)$ on $H_\pi$ by $$\pi(f) v = \int_G f(g) \pi(g)(v) d\mu(g).$$

Let us say here that $f \in L^1(G)$ is of trace class if for every unitary, absolutely irreducible representation $\pi$ of $G$, the operator $\pi(f)$ is of trace class. (side question: is there a name outside for this notion?).

I have many question about this notions, but let me give one:

Is it true that the space of trace class functions on $G$ is dense in $L^1(G)$? If not, is it true for a separable type I locally compact group ?

I know of certain larges classes of group for which it is true. Compact group and abelian locally compact groups are trivial examples, since all irreducible $\pi$'s are finite dimensional. Real and $p$-adic Lie group are other examples since smooth compact support functions are trace class (IIRC) with the usual meaning of smooth ($C^\infty$ in the real case, locally constant in the $p$-adic case), and are dense. Yet this is proved in a rather ad hoc way, for example in the real case (Duflo-Labesse) by showing that such a function $f$ is the convolution of two such functions $f_1$ and $f_2$, so that $\pi(f)=\pi(f_1)\pi(f_2)$ and $\pi(f_1), \pi(f_2)$ are Hilbert-Schmiddt. Also the adelic case (necessary for the global trace formula) follows from the real and $p$-adic case.

Yet I don't know if this is true in general, or at least for a large abstract class of locally compact group (defined with a property such as type I, not as a list of example such as "real Lie groups"), with if possible a uniform proof.

If the question is too difficult, or the answer is no, does that help if we ask the same question for functions that are reduced trace class, i.e. such that $\pi(f)$ is of trace class for all irreducible $\pi$'s in the reduced spectrum (i.e. in the support of the regular representation).

PS: Clearly, the condition of being a trace class does not change if we replace $L^1(G)$ by $C^\ast(G)$ with its natural norm. Hence one of the tag.

Answer 1

The following is a suggestion how to prove a weak variant of the OP's original question affirmative.

Theorem: Let $G$ be a unimodular, seperable, type-I group. For every element $\phi$ in $C_c^\infty(G)$ (in the sense of Bruhat), the operator $\pi(\phi)$ is trace class for almost every unitary representation $\pi$ (with respect to the Plancherel measure).

Proof: The Plancherel theorem states for $\phi \in C_c^\infty(G)$, that $\pi( \phi \ast \phi^{\ast})$ is a trace class operator for a.e. $\pi$. The Dixmier-Malliavin theorem allows us to write general $f$ as a sum of convolution product $S \ast T^*$. A convolution product is the linear combination of four positive functions (polarisation identity), i.e.,

$$4 \cdot S \ast T^\ast = (S+T) \ast (S+T)^\ast - (S-T) \ast (S-T)^\ast$$ $$ - i(S-iT) \ast (S-iT)^* + i(S+iT) \ast (S+iT)^*.$$ Applying that $ \pi ( (S+T) \ast (S+T)^\ast ), \dots$ are trace class by Plancherel for a.e. $\pi$ yields the theorem. Q.e.d.

The following lemama should follow rather easily from the equivalent result of Lie groups:

Lemma: $C_c^\infty(G) \subset L^1(G)$ is a dense embedding.

Discussion of Dixmier-Malliavin: Every smooth, compactly supported functions is a finite sum of convolution products. This is due to Dixmier and Malliavin, originally proven for a Lie groups only.

Writing general $G$ as an projective limit of Lie groups, we obtain a notion of smooth, compactly supported functions on $G$. So the proof theorem Dixmier-Malliavin extends to locally compact groups. See also my Phd thesis http://webdoc.sub.gwdg.de/diss/2012/palm/palm.pdf, where I discuss this in chapter 5, in particular Theorem 5.2.1. Approximation via Lie groups is explained in chapter 4.

Discussion of the Plancherel theorem: A necessity of the Plancherel theorem (check e.g. http://www.encyclopediaofmath.org/index.php/Unitary_representation) is that the $tr\; \pi (f\ast f^*)$ is well-defined for every $f \in L^1(G) \cap L^2(G)$ and a.e. $\pi$ with respect to the Plancherel measure $\mu$. The Plancherel theorem states then for $G$ be a unimodular separable locally compact group of type I $$ \int\limits_G |f(g)|^2 d g = \int\limits_{unitary \; dual} tr\; \pi( f \ast f^*) d \mu(\pi).$$ There exist extensions, which drop unimodular, type I or seperable as stated in the above link, but don't know how the extensions look like, so that's why I worked with rather special $G$ here.

You can furthermore put a (Fell-)topology on the unitary dual and so on, but I don't know wheter $\pi \mapsto \pi(\phi)$ will then be a continuous function!? Even a smooth structure via the approximation-by-lie-groups-business.

Using Dixmier-Malliavin plus the polarization identity you obtain for $f \in C_c^\infty(G)$ (Theorem 6.2.6, pg.83 in my thesis) the following variant of the Plancherel theorem $$ f(1) = \int\limits_{unitary \; dual} tr\; \pi( f) d \mu(\pi).$$

For non-type-I groups, the Plancherel measure will not be unique (?), so I am sceptical about a direct generalization omitting that notion. I think that neither being unimodular nor seperable is a crucial hypothesis.

Discussion of reductive groups: For reductive groups over local fields, you can replace a.e. w.r.t. to Plancherel measure by every smooth, admissible representation, but I guess you are well aware of that. Some reps are not tempered and do not contribute to the Plancherel formula, so this far more. The notion of smooth in the sense I indicate above coincides with the usual one in the non-archimedean world/totally disconnected groups. Be careful, general Lie groups are not type I. For the adelic points of a reductive group over a global field, you need an argument along the lines of Is a reductive adelic group a Type I group?, but the situation is similar to that of reductive local groups.