Do all unitary representations weakly converge to zero at infinity?

Question

Question. Let $G$ be a non-compact, finite dimensional Lie group, and let $(X, \mu)$ be a Radon measure space. Let $$\rho\colon G\to U(L^2(X))$$ be a unitary, strongly continuous, representation. Is it true that, if $g_n\to \infty$, then $$ \int_X \overline{h(x)}\rho_{g_n} f(x)\, d\mu\to 0, \qquad \forall f, h\in L^2(X)?$$ Appropriate hypotheses on $X$ may be assumed.

Here, $g_n\to \infty$ means that, for any compact $K\subset G$, $g_n\notin K$ for all sufficiently big $n\in\mathbb N$.

EDIT (for reference). I learned in the comments that a fairly complete answer is given by the "Howe-Moore vanishing theorem". I found a reference in this book (Bekka - Mayer, "Ergodic theory and topological dynamics of group actions on homogeneous spaces"); it is Theorem 1.1 at page 81.

---

This property is true in the following cases.

1. $G=(\mathbb R^n, +)$, $X=\mathbb R^n$ with Lebesgue measure and $\rho_g f(x):=f(x-g)$.
2. $G=(\mathbb R_{>0}, \cdot)$, $X=\mathbb R^n$ with measure $d\mu=\frac{dx}{\lvert x\rvert^n}$, and $\rho_g f(x):=f(x/g)$.
3. $G=SU(1, 1)$, $X=\mathbb D$, the unit disk, with measure $d\mu=\frac{4dxdy}{(1-(x^2+y^2)^2)^2}$, and $$\rho_g f(z):=f\left(\frac{az+b}{\overline b z + \overline a}\right), \qquad g=\begin{bmatrix} a & b \\ \overline b & \overline a\end{bmatrix}, $$ where $|a|^2-|b|^2=1$.
4. $G=SL(2, \mathbb R)$, $X=\mathbb H=\{z\in \mathbb C\ :\ \Im z>0\}$, with measure $d\mu=\frac{1}{y^2}dxdy$, and $$\rho_g f(z):=f\left(\frac{az+b}{c z + d}\right), \qquad g=\begin{bmatrix} a & b \\ c & d\end{bmatrix}, $$ where $ad-bc=1$. This case is actually isomorphic to the previous one, via the Cayley transform $z\mapsto \frac{z-i}{z+i}$ that maps $\mathbb H$ onto $\mathbb D$.

I learned the proof for the example 2 in this Math.SE post. The same idea works for the other examples, and it is even slightly simpler; indeed, in all three cases, for all $f\in L^2(X)$, and for all compact $A\subset X$, $$\lVert \rho_{g_n} f\rVert_{L^2(A)}\to 0, $$ provided that $g_n\to \infty$.$^{[1]}$ Thus, $$ \left\lvert \int_X \overline{h(x)}\rho_{g_n}f(x)\, d\mu\right\rvert \le \lVert h\rVert_{L^2(A)}\lVert \rho_{g_n}f\rVert_{L^2(A)}+ \lVert h\rVert_{L^2(X\setminus A)}\lVert \rho_{g_n}f\rVert_{L^2(X\setminus A)}. $$ The first summand tends to zero, while the second can be made arbitrarily small by choosing a sufficiently big $A$, because $h\in L^2(X)$. Here we use that $\rho$ is unitary.

---

$^{[1]}$ As mentioned, this case is slightly simpler than the dilation one, because for the dilation group we must also consider the possibility that the $L^2$ norm concentrates at the origin.

Answer 1

No, there are simple counterexamples. E.g., take $G = \mathbb{R}$ and $X = \mathbb{C}$ with Lebesgue measure, and define $\rho_t f(z) = f(e^{2\pi i t}z)$ for $t \in \mathbb{R}$ and $f \in L^2(\mathbb{C})$. Then $\rho_t$ is the identity for any integer $t$, so $\rho_n \to {\rm id}$ strongly, not to zero.

Answer 2

In the comments to Nik's answer, Nate Eldredge and Paul Garrett point out that the answer is not "truly noncompact"; indeed, in that answer the noncompact group $(\mathbb R, +)$ factors through $\mathbb R/\mathbb Z$. However, we can easily fix this and construct a "truly noncompact" counterexample, by adding a toroidal component with a dense linear flow.

Precisely, let $X=\mathbb T^2\times \mathbb C$ with Lebesgue measure, where $\mathbb T^2=\mathbb R^2/\mathbb Z^2$. Let $v=(\alpha, \beta)\in\mathbb R^2$ be such that $$\tag{1}\frac\beta\alpha\text{ is irrational, }$$ and define a unitary representation of $(\mathbb R, +)$ by $$ \rho_t f(x, z):=f(x-tv, e^{-it}z), \qquad \forall f\in L^2(\mathbb T^2\times \mathbb C).$$ Since there is $t_n\to \infty$ such that, say, $\lvert t_n v\rvert\le \frac1{100}$, this representation does not vanish at infinity, just like in Nik's answer; but now there is no compact quotient of $\mathbb R$ that $\rho$ factors through.

Remark. I considered $\mathbb T^2\times \mathbb C$, instead of just $\mathbb T^2$, to avoid the objection that $\mathbb T^2$ is compact. So, in this example, we have a non-compact group acting on a non-compact space, with an action that cannot be reduced to the one of a compact quotient. This is what I mean by "truly noncompact".