Question: Is there a linear map $\mathcal F$ from the Hilbert space of $\ell^2$ functions on the discrete Heisenberg group to some Hilbert space of functions $ L^2(\bigcup \{\Omega_n\}) $, such that:
$\mathcal{F}(f *g)|_{\Omega_n}=\mathcal{F}(f)|_{\Omega_n} {\cdot } \mathcal{F}(g)|_{\Omega_n}$ for some non-commutative product $\cdot $,
$\langle f,g \rangle=\langle \mathcal{F}(f),\mathcal F(g) \rangle$, i.e. the Plancherel formula holds, and
The spaces $\Omega_1$, $\Omega_2$, ... are as small as possible?
Motivation: This is the setting of a non-commutative circle method that could be used in number theory. Specifically, the problem I am investigating is whether all odd numbers $q$ can be written in the form $wx-xy+yz$ for $w$, $x$, $y$, $z$ primes.
I used the discrete Heisenberg group as an encoding, i.e. let $A$ be the set of matrices $$ \begin{bmatrix} 1 & a & 0 \\ 0 & 1 & b\\ 0 & 0 & 1 \end{bmatrix} $$ $B$ the set of matrices $$ \begin{bmatrix} 1 & a & 0 \\ 0 & 1 & -b\\ 0 & 0 & 1 \end{bmatrix} $$
where $a$ and $b$ are primes, and $C$ the set of all matrices with entry $C_{1,3}=q$.
Let $I_A$ be the indicator function of $A$ times some cutoff function to make $I_A$ an $\ell^2$ function, and similarly are $I_B$ and $I_C$. Then the problem is equivalent to computing the inner product $\langle I_A * I_B * I_A, I_C \rangle$.
Remark: If the discrete Heisenberg group were finite, we could take the $\Omega$s to be the irreducible representations of the group, and $\cdot$ the product of matrices in the irreducible representations. Then all three conditions would be satisfied. It's also the case when the discrete Heisenberg group is replaced by the integers, i.e. the original circle method, though the $L^2$ norm is defined differently.
