I am trying to find the matrix coefficients $\{\hat{c}^{\alpha}_{i j}\}$ that minimize the mean squared error against a function $f(g)$ over a compact group to some bandwidth cutoff $\ell$.
$$\DeclareMathOperator*{\argmin}{arg\,min} \argmin_{\{\hat{c}^{\alpha}\}} \int_{\mathcal{G}} [ f(g) - \sum_{\alpha}^{\ell} \mathrm{Tr}\ \hat{c}^{\alpha} \rho^{\alpha}(g) ]^2\ dg$$
$\{\rho^{\alpha}(g)\}_{\alpha = 0}^{\ell}$ are irreducible, orthogonal group representations indexed by $\alpha$. Setting the partial derivatives of the above expression to zero yields
$$\int_{\mathcal{G}} f(g) \rho^{\beta}_{m n}(g)\ dg = \sum_{\alpha}^{\ell} \sum_{i,\ j} \hat{c}^{\alpha}_{i j} \int_{\mathcal{G}} \rho^{\alpha}_{j i}(g) \rho^{\beta}_{m n}(g)\ dg$$
When integrating over the entire group $\mathcal{G}$, because of the Schur orthogonality relations, this simplifies to
$$\int_{\mathcal{G}} f(g) \rho^{\beta}_{m n}(g)\ dg = \hat{c}^{\beta}_{n m} \frac{1}{d^{\beta}}$$
and the resulting $\{\hat{c}^{\alpha}\}_{\alpha = 0}^{\ell}$ are just the Fourier coefficients. However, instead of integrating over the entire group $\mathcal{G}$, I only want to integrate over group elements that satisfy a certain condition, for instance
$$\mathcal{G}' = \{g \in \mathcal{G} \mid a < \mathrm{Tr}\ M \rho^{1}(g) < b\}$$
This is problematic, however, because finding the $\{\hat{c}^{\alpha}\}_{\alpha = 0}^{\ell}$ now entails a computationally expensive number of integrals since
$$\int_{\mathcal{G'}} \rho^{\alpha}_{j i}(g) \rho^{\beta}_{m n}(g)\ dg \neq \delta_{\alpha \beta} \delta_{j m} \delta_{i n} \frac{1}{d^{\beta}}$$
Is there a way I can ameliorate this situation?
