# Is the following set convex or not?

Let $(E, \langle\cdot\;, \;\cdot\rangle)$ be a complex Hilbert space. Let $T\in\mathcal{L}(E)$ and $M\in \mathcal{L}(E)^+$.

Assume that $T(ker(M))\nsubseteq ker(M)$. We define the following subset:

\begin{eqnarray*} S_M(T) &=&\{\lambda\in \mathbb{C}\,;\;\; \exists\,(\alpha_n,\beta_n)\in ker(M)\times \overline{Im(M)}\,;\;\;\|M^{1/2}\beta_n\|=1, \displaystyle\lim_{n\rightarrow+\infty}\langle MT \alpha_n\; |\;\beta_n\rangle+\langle MT \beta_n\; |\;\beta_n\rangle=\lambda,\\ &&\phantom{+++++}\;\hbox{and}\;\displaystyle\lim_{n\rightarrow+\infty}\|M^{1/2} T(\alpha_n+\beta_n)\|<\infty\;\}. \end{eqnarray*} What do you think about the convexity of $S_M(T)$?? I try with an example of $M$ and $T$ such that $T(ker(M))\nsubseteq ker(M)$, I get $S_M(T)=\mathbb{C}$.

I claim that $S_M(T)=\mathbb{C}$. Do you think that my claim is true?

• I am curious about what $T$ stands for. You promised the definition of a semi-definite sesquilinear form, but I am unable to find it. Nov 8 '17 at 16:20
• Isn't it rather well known that $S_M(T)$ is convex when $T = I$ instead of when $M = I$? (Hausdorff-Töplitz Theorem) Nov 9 '17 at 10:25
• When $T=I$, isn't it obvious that we have $S_M(T)=\{1\}$?
– B K
Nov 9 '17 at 17:06

Suppose $M\geq 0$ and $T(\ker M) \nsubseteq \ker M$. So there exists an $\alpha \in \ker M$ such that $T\alpha \notin \ker M$ which gives that $MT\alpha \neq 0$ and since $M$ is positive then $M^{1/2}MT\alpha\neq 0$ as well.
Let $\beta = \frac{MT\alpha}{\|M^{3/2}T\alpha\|} \in {\textrm Im}(M)$ then $\|M^{1/2}\beta\| = 1$ and $\|M^{1/2}T(\alpha + \beta)\| < \infty$. If $\lambda \in \mathbb C$ then $(\lambda \alpha, \beta) \in \ker M \times {\mathrm Im}(M)$ and \begin{align*} \langle MT(\lambda\alpha)\ |\ \beta\rangle + \langle MT\beta\ |\ \beta\rangle &= \lambda\langle MT\alpha\ |\ \frac{MT\alpha}{\|M^{3/2}T\alpha\|}\rangle + \langle MT\beta\ |\ \beta\rangle \\ & \lambda\frac{\|MT\alpha\|}{\|M^{3/2}T\alpha\|} + \langle MT\beta\ |\ \beta\rangle. \end{align*} Because $\alpha$ and $\beta$ are fixed and $\|MT\alpha\| \neq 0$ then this gives you any complex number for an appropriate choice of $\lambda$. Therefore, $S_M(T) = \mathbb C$.
• I think that there exists a problem because $\|M^{1/2}T(\lambda\alpha + \beta)\|$ is not necessary $< \infty$ for an arbitrary complex number $\lambda$. Mar 20 '18 at 8:11
• I think that if we consider $E = \mathbb C^2$, $M= \begin{pmatrix} 1 & 0 \\ 0 & 0 \end{pmatrix}$, $T= \begin{pmatrix} 0 & 1 \\ 0 & 0 \end{pmatrix}$, we get $S_M(T)\neq\mathbb{C}$ Mar 20 '18 at 8:39
• The norm statement in your first comment is always finite by the triangle inequality. The confusion probably arises from the fact that my $\lambda$ and your $\lambda$ play different roles. I've double checked everything and the argument looks good. Try to work through your example above with the recipe I gave. Mar 20 '18 at 14:31