Let $\mathcal H$ be a Hilbert space, and let $a \in \mathcal B(\mathcal H)$ satisfy $\mathrm{Tr}(a)=0$. If $a$ is selfadjoint, then we can find a vector $\xi \in \mathcal H$ such that $\langle \xi  a \xi \rangle=0$. Is this true in general? Is it always possible to find an orthonormal basis of such vectors?

3$\begingroup$ True in finite dimension... $\endgroup$ – Igor Rivin Jan 19 '11 at 2:18
The answers to both your questions are yes. (ie for any trace class operator $a$ on a Hilbert space $H$ with $Tr(a)=0$, there exists an orthonormal basis of $H$ consisting of vectors $\xi$ such that $\langle a \xi,\xi\rangle=0$).
First note that the fact that there exists an orthonormal basis of vectors $\xi$ such that $\langle a \xi,\xi\rangle=0$ is immediate (by something like "transfinite induction") from the fact that for any $a$ with trace zero, there exists one nonzero vector $\xi$ with $\langle a \xi,\xi\rangle=0$. Indeed, Zorn's Lemma implies that there exists a maximal closed subspace $H$ of $K$ such that there is an orthonormal basis of $K$ consisting of vectors such that $\langle a \xi,\xi\rangle=0$. If $K$ were different from $H$, consider $P$ the orthonormal projection on the orthogonal of $K$. The restriction of $Pa$ to the orthogonal of $K$ is still of trace class and zero trace, so there exists a unit vector in the orthogonal of $K$ such that $\langle a \xi,\xi\rangle=0$, which contradicts the maximality of $K$.
Now to the first point. You want to prove that is $Tr(a)=0$, zero belongs to the numerical range $W(a)$ of $a$, which is the set of $\langle a \xi,\xi\rangle$ for $\xi$ in the unit sphere of $H$. $W(a)$ is a convex subset of $\mathbb C$ for any operator $a$. This is usually stated for matrices (see this page), but since everything happens in two dimensions this remains true in general.
But the assumption $Tr(a)=0$ implies that there is a sequence $(\lambda_n)_n$ in $W(a)$ such that $\sum_n \lambda_n=0$. This clearly implies that $0$ belongs to the convex hull of the $\lambda_n$'s, which is contained in $W(a)$.
Edit: To answer Bill's objection, here is a proof that if $\sum_n \lambda_n=0$, then $0$ belongs to the convex hull of the $\lambda_n$'s. (I agree that for a general compact operator, $0$ does not always belong to $W(a)$ but only to its closure, but my point was that here the assumption $Tr(a)=0$ makes it different).
Assume, by contradiction, that $0$ does not belong to the convex hull of the $\lambda_n$'s. By (some form of) HahnBanach, the $\lambda_n$'s all belong to some closed halfplain, say for simplicity $Im(\lambda_n)\geq 0$ for all $n$. The equality $\sum_n Im(\lambda_n)=0$ then implies that the $\lambda_n$'s are all real. They cannot be all positive (or all negative), otherwise $\sum_n \lambda_n \ne 0$. This is a contradiction.

$\begingroup$ Mikael, you only get that $0$ is in the closure of the convex hull of the $\lambda_n$'s. IIRC, the numerical range need not be closed. For those who don't know: in finite dimensions Mikael's argument works and is the standard proof that an $n$ by $n$ matrix that has trace zero is unitarily equivalent to a matrix that has zero diagonal. Of course, here simple induction is used. $\endgroup$ – Bill Johnson Jan 19 '11 at 23:10

$\begingroup$ Bill: unless I miss something, I really believe that my argument proves that the condition $Tr(a)=0$ implies that the numerical range of $a$ is closed. See my edit. $\endgroup$ – Mikael de la Salle Jan 20 '11 at 8:28

$\begingroup$ Mikael, I'm confused with a couple of things in your answer. First you seem to imply that an operator with zero trace is traceclass (and thus compact), which is not the case. Second, in your maximality argument in the second paragraph you say that the compression of a tracezero operator by a projection is still tracezero, which again I think is not true. $\endgroup$ – Martin Argerami Jan 21 '11 at 19:40

$\begingroup$ @Martin: How do you define the trace of an operator which is not of trace class? For you second remark, what I meant is that if the compression to a closed subspace $K$ of a tracezero operator has trace zero, then so has its compression to the orthogonal of $K$. This is just because of the equality $Tr(PaP+(1P)a(1P))=Tr(a)$, valid for any projection $P$. $\endgroup$ – Mikael de la Salle Jan 21 '11 at 22:02

$\begingroup$ Thanks a lot, Mikael! I should start thinking before writing... $\endgroup$ – Martin Argerami Jan 22 '11 at 2:05
The Igor's answer also works if $a$ is compact selfadjoint and $\mathcal{H}$ is isomorphic to $\ell^2(\mathbb{N})$ .
Because of the spectral theorem and $\text{Tr}(a)=0$, for any $v\in \mathcal{H}$ we have :
$$ \langle v,av\rangle =\sum_{i=1}^{\infty}\lambda_i v^2_{r_i} \sum_{i=1}^{\infty}\beta_i v^2_{s_i}\qquad (1) $$
where $(\lambda_i)_{i\in\mathbb{N}}\neq (0,0,\ldots)$ and $(\beta_i)_{i\in\mathbb{N}}\neq (0,0,\ldots)$ and for all $i$ we have $\lambda_i,\beta_i\geq 0$. Here $\lambda$'s and $\beta$'s form the spectrum of $a$.
Note that $\ell^2(\mathbb{N})=V_+\oplus V_{}$, where the spaces $V_{+}=\bigoplus_{i=1}^{\infty}e_{r_i}$ and $V_{}=\bigoplus_{i=1}^{\infty}e_{s_i}$
We can rewrite (1) as
$\pi_{1}(v)$ $\pi_{2}(v)=0,$
where the norms appearing above are defined on the subspaces $V_{+}$ and $V_{}$ by two positive bilinear forms associated to the $\lambda$'s and $\beta$'s respectively and $\pi_1$ and $\pi_2$ are projections on the subspaces $V_{+}$ and $V_{}$ .
Taking an orthonormal sets in both spaces in the unit ball, we can find the vectors you want. I guess this idea can be generalized using the functional calculus.

2$\begingroup$ I am interested specifically in nonselfadjoint operators $a \in \mathcal B(\mathcal H)$. Thank you for your idea of obtaining positive bilinear forms from $a$. $\endgroup$ – Andre Jan 19 '11 at 9:46