Induced representation of locally compact groups

I am not having the best luck getting this, so any advice is appreciated. I apologize in advance if the question is too low-level. Let $G$ be a locally compact group $\sigma$-compact group, and $H\leq G$ a closed subgroup such that $G/H$ carries a finite invariant measure $\mu$. Let $(\pi,\cal H_\pi)$ be a unitary representation of $H$, and let $\rho={\rm Ind}_H^G(\pi)$, and call the Hilbert space it acts on by left translations ${\cal H}_\rho$. This space can be realized as the Hilbert space completion of the space of functions $G\to\cal H_\pi$ which are $H$-equivariant, with support compact modulo $H$, with respect to the inner product $$\langle f_1,f_2\rangle=\int_{G/H} \langle f_1(g),f_2(g)\rangle d\mu(gH).$$ This is the standard simplest construction. My question is: how can I define a bounded linear map $\Phi:{\cal H}_\pi\to{\cal H}_\rho$ which is an isometry and commutes with orthogonal projections onto subspaces of invariant vectors.

What I tried is to let $\xi\in{\cal H}_\pi$ and define the function $f_\xi$ by $$f_\xi(x)=\begin{cases}\pi(x^{-1})\xi&\text{if}~x\in H,\\0&\text{otherwise.}\end{cases}$$ This defines a map $\Phi:\cal H_\pi\to\cal H_\rho$, $\xi\mapsto f_\xi$. But then, what I obtain is $$\langle \Phi(\xi),\Phi(\eta)\rangle_{\cal H_\rho}=\mu(H)\langle \xi,\eta\rangle_{\cal H_\pi},$$ with $\mu(H)$ the measure of the identity coset in $G/H$, since $f_\xi,\,f_\eta$ are non-trivial only on $H$. My map seemed fairly natural, is there something I am overlooking? Is there some twist to apply to this to make it work?

Such a $\Phi$ is not going to exist — nor is $H$ going to have nonzero $\mu(H)$ — unless $H$ is open as well as closed in $G$. This is the very case treated in Mackey (1951, Part II): then $G/H$ is discrete, $\mu$ can be normalized to be counting measure (i.e. $\mu(eH)=1$), and then your $\Phi$ is precisely the isometric embedding $\mathcal H_\pi\hookrightarrow\mathcal H_\rho$ already discussed and denoted $v\mapsto f_{e,v}$ by Mackey (p. 585).
(When $H$ is not open one resorts to more sophisticated maps, e.g. the $\epsilon:\mathrm C_0(G)\times\mathcal H_\pi\to\mathcal H_\rho$ of Blattner (1962): $$\epsilon(\phi,\xi)(g)=\int_H\phi(hg)\pi(h^{-1})\xi\,dh,$$ assuming here, as you do, that $G/H$ has a $G$-invariant measure, i.e. the modular functions of $G$ and $H$ coincide on $H$.)
• Dear @Francois, thank you very much for the detailed answer and references. If I may ask a few follow-up questions: is your $C_0(G)$ the space of continuous functions with compact support on $G$? Concerning your second remark, this is a standard way to obtain an equivariant function, but are you saying that this should define an isometric embedding, in some special case? I guess the case that I am ultimately interested in is that of a lattice... – Math-user Dec 2 '16 at 4:01
• @Math-user About $\mathrm C_0(G)$: yes. About isometry: no, as $\epsilon$'s domain has no metric for which this would make sense; but Blattner (1961, p.82) shows $\epsilon$'s continuity in the appropriate inductive limit topology, i.e. $\operatorname{supp}\phi\subset K$ implies $\|\epsilon(\phi,\xi)\|\leqslant\Lambda_K\sup|\phi|\|\xi\|$ for some constant $\Lambda_K$. – Francois Ziegler Dec 2 '16 at 4:32
• Dear @Francois, one remark/question: if you assume that $G/H$ is discrete, you would want $\mu(G/H)=1$, not $\mu(eH)=1$, since that would not be invariant right? But there is no reason why $G/H$ should be compact/finite, and in that case, $\mu(G/H)$ cannot be finite, in which case the construction I gave does not work? For the isometry question in my comment, I guess what I asked is: if I take $\phi$ to be an approximate identity, can I use it to define an isometric embedding? – Math-user Dec 2 '16 at 22:11
• @Math-user Your construction (or Mackey's) works whether or not $G/H$ is compact (or $\mu$ finite), but only if $G/H$ is discrete ($\Leftrightarrow\mu=\mathrm{const}\cdot\mathrm{counting}$; this is invariant!). As you noted, it yields an embedding which is isometric up to the same $\mathrm{const}$ factor. When $G/H$ is not discrete, you might sometimes embed (e.g. $G/H=\mathbf{R}/\mathbf{Z}$ and trivial $\mathcal H_\pi=\mathbf C\simeq$ constants in $\mathcal H_\rho$), but not by using $H$-supported functions — nor, I believe, an approximate identity $\phi\to$ Dirac at $eH$. – Francois Ziegler Dec 3 '16 at 1:23