Here are two setup where the notion of "mildly mixing" comes up: for representations and for group acting by measure preserving transformations (see definitions below). Since a natural class of representations come from such actions, the question is:
$\mathbf{Question:}$ are these notions equivalent?
For definitions and details: in both case, $\Gamma$ is some [finitely generated] discrete group. The classical way to make a measure-preserving action into a Hilbertian representation (the Koopman representation) is to consider $\Gamma$ acting by translation on the functions in $L^2_0(X,\mu) =: \mathcal{H}$ (the index $0$ means one restricts to functions orthogonal to the constant function; if the constant function is in $L^2$).
(a) if $\Gamma$ acts by measure preserving transformations on a standard probability space $(X,\mu)$, then the action is called mildly mixing if $$ \forall A \subsetneq X \text{ with } A \neq \emptyset, \liminf_{g \to \infty} \mu(A \triangle g A) >0 $$ where $g \to \infty$ means for any sequence of elements which leaves all finite sets, see Ch2.§10.¶(A) p.62 of "Global aspects of ergodic group actions" by Kechris.
(b) If $\pi$ is a Hilbertian representation of $\Gamma$ (on the Hilbert space $\mathcal{H}$), it is mildly mixing if $$ \forall \xi \in \mathcal{H} \setminus \lbrace0\rbrace, \liminf_{g \to \infty} \| \pi(g) \xi - \xi \| >0 $$ see $\mathit{ibid.}$ p.217 in App.H.¶(D). One can reformulate this as $\limsup_{g \to \infty} \langle \pi(g) \xi \mid \xi \rangle < 1$.
$\mathbf{Question~1:}$ It is fairly easy to see that (b)$\implies$(a). But does anyone has a reference for (a)$\implies$(b)? [Edit: there is a proof in "Asymptotic properties of unitary representations and mixing" by K. Schmidt.]
There is another definition of mildly mixing (from the book "Ergodic theory via joinings" by Glasner, Ch3.1 Definition 3.2 p.63 and Ch8.§5 Definition 8.15 p.169). Here, $\Gamma$ acts by measure preserving homeomorphisms on the compact space $X$ and the action is ergodic
(a') the action is mildly mixing iff there are no rigid factors. $\Gamma \circlearrowleft (X,\mu)$ is called rigid if the topology induced on $\Gamma$ by the Koopman representation and the strong operator topology is not discrete.
Note that in (a') the action cannot have a kernel whereas in (a) or (b) it could have a finite kernel.
$\mathbf{Question~2:}$It is not too hard to see that, if there is no kernel, (b) implies (a') [actually, not(a') implies not(b)]. But what about the converse?
Stangely, in Glasner's book, there is an exercise (Ch8.§5, exercise 8.17 p.169) saying (a')$\iff$(b) for Abelian groups (this proof comes from a paper of Lemanczyk and Lesigne). But there is no mention towards the fact it may or may not hold in greater generality...
It is also clear that (b) implies
(b') the kernel of $\pi$ is finite and there are no sub-representation $\tilde{\pi}$ so that $\tilde{\pi}(G)$ is not discrete in $U(\mathcal{\tilde{H}})$ for the strong operator topology (the analogue of "no rigid factors").
$\mathbf{Question~3:}$What about (b')$\implies$(b)?
Again, this converse is simple in the Abelian case.
So to sum up one has $$ \begin{array}{ccc} (b) & \iff & (a) \\ \Downarrow & & \Downarrow \\ (b') & \implies & (a') \end{array} $$ and the $\Downarrow$s are only known to be reversible when the groups is Abelian. (a') $\implies$(b') might be a generic argument...
