Recently I have just started to work on the trace formula of the metaplectic representation of ${\rm Sp}(2n,{\mathbb C})$ group. I have a naive question: is the trace (i.e., character of the representation) always nonzero? Why?
Thanks a lot for any help.
I've deleted my previous answer, since I badly misinterpreted the question.
The metaplectic representation $\pi$ of $\operatorname{Sp}(2n,\mathbb C)$ is the direct sum of two infinite dimensional irreducible unitary representations. As Speyer points out $\pi(g)$ is not of trace class. However, by Harish Chandra's theory, the global character of $\pi$, which is a distribution, is represented by a unique, smooth, conjugation invariant funtion $F_\pi$ on the set of regular semisimple elements. So it makes sense to ask if $F_\pi(g)\ne 0$ for all regular semisimple elements. The answer is yes.
Over any local field other than $\mathbb C$ there is a formula, due to Roger Howe:
$$\lvert F_\pi(g)\rvert=\lvert\det(1-g)\rvert^{-1/2}.$$
Note that $g$ is in the metaplectic group, but $|F_\pi(g)|$ factors to the symplectic group. In some sense the formula is true without the absolute values up to an $8^\text{th}$ root of unity.
For the real case see [Adams, Israel Journal, 1997, Proposition 4.7]. The complex case, which is easier, is
$$\lvert F_\pi(g)\rvert=\lvert\det(1-g)\rvert^{-1}.$$
See Section 6 of the same paper.
It's possible that I should have waited for someone else to answer this. As you'll see, there are a lot of things that I am confused by. Nonetheless, I'm going to see what I can work out.
The metaplectic representation of $\text{Sp}_{2n}(\mathbb{R})$ is infinite dimensional, so a careful answer to this question would address whether these operators are trace class or, if not (I suspect "not"), in what sense the trace is defined. I'm bad at analysis, so I'll just answer the formal question. The answer will be that the thing which should be the character of the metaplectic representation is never $0$, though it is sometimes $\infty$.
Inside $\text{Sp}_{2n}(\mathbb{R})$ there is a copy of $\text{SL}_2(\mathbb{R})^n$, and every element of $\text{Sp}_{2n}(\mathbb{R})$ with distinct eigenvalues is conjugate to an element of $\text{SL}_2(\mathbb{R})^n$. Whatever trace means, it should certainly be conjugacy invariant, so I just need to figure out what trace means on $\text{SL}_2(\mathbb{R})^n$. And the metaplectic representation on $\text{SL}_2(\mathbb{R})^n$ is some sort of completion of the tensor product of the metaplectic representations of the individual $\text{SL}_2(\mathbb{R})$ factors, so we just need to do the $\text{SL}_2(\mathbb{R})$ case and then multiply them together.
I first attempted this by thinking about the connected component of the split torus in $\text{SL}_2(\mathbb{R})$; matrices of the form $\left[ \begin{smallmatrix} r&0 \\ 0& r^{-1} \end{smallmatrix} \right]$ for $r>0$. This acts on $L^2(\mathbb{R})$ by $f(x) \mapsto \sqrt{r} f(rx)$. Ignoring that we are supposed to use functions in $L^2$, the eigenfunctions are $f(x) = x^k$, with eigenvalues $r^{k+1/2}$. If we sum just over nonnegative integer $k$, we get that the trace should be $\sum_{k=0}^{\infty} r^{k+1/2} = \tfrac{r^{1/2}}{1-r} = \tfrac{1}{r^{-1/2} - r^{1/2}}$.
I remark that the corresponding Lie algebra action is that the Lie algebra element $\left[ \begin{smallmatrix} 1&0 \\ 0& -1 \end{smallmatrix} \right]$ acts by $x \tfrac{d}{dx} + 1/2$ (differentiate $\sqrt{r} f(rx)$ with respect to $r$ at $r=1$); this matches the formula (5.17) here.
That was pretty sketchy, so let's try the compact torus $\left[ \begin{smallmatrix} \cos \theta & - \sin \theta \\ \sin \theta & \cos \theta \end{smallmatrix} \right]$ instead.
This is the exponential of $\theta J$, where $J = \left[ \begin{smallmatrix} 0 & - 1 \\ 1& 0 \end{smallmatrix} \right]$. According to (5.17) here, $J$ acts by $\tfrac{1}{2} (x^2 - \tfrac{d^2}{(dx)^2})$, and you compute the eigenvalue of $J$ in any first course on quantum mechanics -- they are $k+1/2$ for $k \in \mathbb{Z}_{\geq 0}$. So the eigenvalue of $\exp(\theta J)$ should be $e^{i \theta (k+1/2)}$ and the trace of $\exp(\theta J)$, in some sense, should be $\sum_{k=0}^{\infty} e^{i (k+1/2) \theta} = \tfrac{e^{i \theta/2}}{1-e^{i \theta}} = \tfrac{1}{e^{-i \theta/2} - e^{i \theta/2}}$.
We've now done the computation in two sketchy ways, and they both gave the same answer: An element of $\text{SL}_2(\mathbb{R})^n$ with eigenvalues $t_1$, $t_1^{-1}$, ..., $t_n$, $t_n^{-1}$ should act with trace $\prod_j \tfrac{1}{t_j^{-1/2} - t_j^{1/2}}$.
I'll now stop trying to do analysis all together, and just show that $\prod_j \tfrac{1}{t_j^{-1/2} - t_j^{1/2}}$ extends to an analytic, nowhere vanishing (but sometimes infinite) class function on the whole metaplectic group, including the parts not conjugate to $\text{SL}_2(\mathbb{R})^n$.
Let $\Delta(t_1, \ldots, t_n) = \prod_i (t_i^{1/2} - t_i^{-1/2})$. So $\Delta^2$ is a well defined function on the diagonal symplectic matrices. I claim that $\Delta^2$ extends to a well defined class function on $\text{Sp}_{2n}$ and thus $\Delta$ to a well defined function on the metaplectic cover. In particular, $\Delta$ does not blow up to $\infty$, so $\tfrac{1}{\Delta}$ is never $0$. (However, $\Delta = 0$ whenever one of the $t_i$ is $1$, so $\tfrac{1}{\Delta}$ can be $\infty$; for example, this happens at the identity.)
To see that $\Delta^2$ is a well defined function on $\text{Sp}_{2n}$, note $\Delta^2$ is a Laurent polynomial which is invariant by the type $C$ Weyl group. (That is to say, by $S_n \ltimes \{ \pm 1 \}^n$, acting by permuting and inverting the $t_i$.) So $\Delta^2$ is a linear combination of characters of finite dimensional representations of $\text{Sp}_{2n}$, and thus extends to a class function on all of $\text{Sp}_{2n}$. For example, if $n=1$, then $$\Delta^2 = (t^{1/2} - t^{-1/2})^2 = t - 2 + t^{-1} = \chi_V - 2 \chi_{\mathbb{1}}$$ where $V$ is the standard representation of $\text{SL}_2$ and $\mathbb{1}$ is the trivial representation.
To see that the square root, $\Delta$ is well defined on the metaplectic cover, we need to remember how to think about $\pi_1(\text{Sp}_{2n}(\mathbb{R}))$. (As I commented above, you say $\mathbb{C}$, but I think you mean $\mathbb{R}$.) The symplectic group $\text{Sp}_{2n}(\mathbb{R})$ contains $\text{SL}_2(\mathbb{R})^n$ and therefore contains the torus $T:=\text{SO}_2(\mathbb{R})^n$. The map $\mathbb{Z}^n = \pi_1(T) \to \pi_1(\text{Sp}_{2n}(\mathbb{R})$ is surjective. The metaplectic cover corresponds to the image of $\{ (c_1, \ldots, c_n) : \sum c_i \equiv 0 \bmod 2 \}$.
So we must show that, if we follow a path through $T$ corresponding to $(c_1, \ldots, c_n)$ with $\sum c_i$ even, then $\Delta$ comes back to itself. Write $(\theta_1,\ldots, \theta_n)$ for coordinates on $T = (\mathbb{R}/2 \pi \mathbb{Z})^n$. The $t_j$ are $e^{i \theta_j}$, so we have $$\Delta(\theta_1, \ldots,\theta_n)^2 = \prod_j (e^{i \theta_j} - 2 + e^{- i \theta_j}) = \prod_j (2 \cos \theta_j - 2) = \prod_j (-4 \sin^2 \tfrac{\theta_j}{2} )$$ so, on the universal cover of $T$, we can write $$\Delta(\theta_1, \ldots,\theta_n) = \prod_j (2 \sqrt{-1} \sin \tfrac{\theta_j}{2} ).$$ If we translate $(\theta_1, \ldots,\theta_n)$ by $(c_1, c_2, \ldots, c_n) (2 \pi)$, then $\prod_j (2 \sqrt{-1} \sin \tfrac{\theta_j}{2} )$ multiplies by $(-1)^{\sum c_i}$. So $\Delta$ is unchanged under translation by $(c_1, c_2, \ldots, c_n)$ with $\sum c_i \equiv 0 \bmod 2$, and thus $\Delta$ is well defined on the metaplectic group.
To reiterate, on the compact torus of the metaplectic group which double covers $T$, the trace of the metaplectic representation should given by $$\frac{1}{\Delta} = \prod \frac{1}{2 \sqrt{-1} \sin (\theta_j/2)},$$ and this formula extends to a well-defined, nowhere $0$ class function on the metaplectic group. However, I am not addressing the question of whether the metaplectic group acts by operators of trace class.
