# On a possible attempt to prove the invariant subspace problem

This question involves a possible method to prove the invariant subspace problem for (separable) infinite dimensional Hilbert spaces. The idea comes from various results on this topic; more precisely, first let me recall one version of Lomonosov Theorem:

Let $$T \in B(H)$$ be a nonscalar bounded linear operator which commutes with some nonzero compact $$K$$. Then, $$T$$ has a nontrivial hyperinvariant (and thus also invariant) subspace.

It is well know that there exist operators for which there is no nonzero compact operator commuting with them (see, for instance, D. W. Hadwin, E. A. Nordgren, H. Radjavi, and P. Rosenthal, An operator not satisfying Lomonosov's hypothesis, J. Funct. Anal. 38 (1980), no. 3, 410-415). The next step is then to consider operators for which there exists at least one nonzero compact $$K$$ such that $$\operatorname{rank} (TK-KT) = 1$$ (the case with rank $$0$$ has been established with Lomonosov result). This result has actually been shown, as for Lomonosov Theorem, in the general setting of Banach spaces. Some references for this are the following ones:

Daughtry, J. An invariant subspace theorem. Proc. Amer. Math. Soc. 49 (1975), 267–268

Kim, H. W.; Pearcy, C; Shields, A. L. Rank-one commutators and hyperinvariant subspaces. Michigan Math. J. 22 (1975), 193-194

Kubrusly, C. S. Hilbert space operators. Birkhauser, Boston, 2003 (Problem and Solution 12.4)

In this case, I wonder if any progress has been done in establishing the existence/nonexistence of operators for which no nonzero compact $$K$$ such that the rank of the commutator is $$1$$ exist. I'm aware of the paper Kim, H. W.; Pearcy, C.; Shields, A. L. Sufficient conditions for rank-one commutators and hyperinvariant subspaces. Michigan Math. J. 23 (1976), no. 3, 235-243, in which some results related to this are obtained, but I would be interested in something more recent, if possible. Furthermore, it has been noted even in the above papers that the case with $$\operatorname{rank} (TK-KT) = 2$$ if established, would prove the invariant subspace problem. Indeed, for rank $$1$$ (compact) operators $$K$$, we always have the rank of the commutator less than or equal to $$2$$. I have tried to follow this way, and this is what I've come up with. First of all, the result does not hold for general Banach spaces, so I have tried to extend a proof in the setting of Hilbert spaces (more precisely, the one in Kubrusly's book Hilbert space operators). Everything can be reproduced verbatim, except the claim:

$$\mathcal{R}(R) \subseteq \mathcal{R}(S)$$

where $$\mathcal{R}$$ denotes the range of the operator, $$S$$ is any operator such that $$0 < \dim \ker S < \infty$$, and $$R:=[T,S]=TS- ST$$. Indeed, the claim is not necessarily true when the rank is equal to $$2$$. To simplify a bit, I will restrict the attention to the case we are actually interested in: in the proof of this statement, following Kubrusly, we will let $$S:= Id - LK$$, where $$Id$$ is the identity operator, $$K$$ is a nonzero compact operator, and $$L$$ is an operator commuting with $$T$$ such that $$\exists g \neq 0 :$$ $$LK g = g$$ (the existence of $$g$$ follows from the assumption that there is no nontrivial invariant subspace, which implies that there is no nontrivial hyperinvariant subspace, and from Lomonosov Lemma). Since we want to deal with rank $$1$$ operators, we can write: $$Kx:= \langle x, z \rangle u$$ for some fixed nonzero $$z,u \in H$$. Now let $$A$$ denote the linear subspace of the points $$x$$ in $$H$$ such that $$TS x \in \mathcal{R}(S)$$. We will start proving that $$R(A)$$, that is, the linear subspace $$R(A):= \lbrace Rx, x \in A \rbrace$$ is equal to $$\operatorname{span} \lbrace STy \rbrace$$ where $$y \in \ker S$$ is such that $$Ty \not \in \ker S$$. The existence of such a point is prove as in Kubrusly's book: it follows from the finite dimensionality of the $$\ker S$$. Since $$\operatorname{rank} R = 2$$ by assumption, this implies that there are points in $$\mathcal{R}(R)$$ not belonging to $$\mathcal{R}(S)$$, thus implying that the above claim is not anymore true when the rank of the commutator is $$2$$. We know that there is a nonzero $$g \in \ker S$$: we can consider any such $$g$$, so let $$g=y$$. Then: $$LKg = \langle y, z \rangle L u = y \Rightarrow Lu = \frac{y}{\langle y, z \rangle}$$ (the denominator is nonzero, otherwise we would have $$L(0) = y \neq 0$$, absurd since $$L$$ is linear). Let $$v \in A$$. Then, this means that: $$TSv = Tv - TLKv = Tv - \langle v, z \rangle TLu = Tv - \frac{\langle v, z \rangle}{\langle y, z \rangle} Ty = St = t - \frac{\langle t, z \rangle}{\langle y, z \rangle} y$$ for some $$t \in H$$. This implies that: $$\langle Tv,z \rangle = \frac{\langle v,z \rangle \langle Ty, z \rangle}{\langle y, z \rangle}$$ We will now show that, for all $$v \in A$$, there is always $$\rho \in \mathbb{C}$$ such that $$Rv=\rho Sty$$, that is, such that: $$TSv= ST(v+ \rho y)$$ Writing again the equation using the expression for $$S$$ given above, we arrive at: $$(\rho + \frac{\langle v, z \rangle}{\langle y, z \rangle}) T( y ) = \frac{\langle T(v + \rho y), z \rangle}{\langle y, z \rangle} y$$ By the previous equation, it is easy to see that: $$\rho= - \frac{\langle v,z \rangle}{\langle y, z \rangle}$$ makes both sides equal to $$0$$ (recall that $$v \in A$$). Thus, $$R(A) = \operatorname{span} \lbrace STy \rbrace$$, and the claim does not anymore hold when the rank is $$2$$. Now, two possible options seem to be available to continue: we either look for another subspace which results to be invariant and closed (here, the closure of the range of $$S$$ would have done this, but the claim does not hold, so we cannot proceed with this space), or we find a contradiction with the space $$A$$ formed this way. I have attempted to follow this second idea. I have shown that $$A$$ is infinite dimensional, and there are infinitely many lineraly independent vectors not belonging to $$A$$. Moreover, $$\forall v, \omega \not \in A$$, $$\exists ! \lambda \in \mathbb{C} :$$ $$(v - \lambda \omega) \in A$$ I do not know if this leads to any contradiction. My question is: has been done some work on the direction that I have shown above? And if the answer is yes, could you please give me some references?

• Interesting post, but I'm skeptical that this approach will work. The existence of a compact operator with ${\rm rank}(TK - KT) \leq 2$ is trivial, so it's unlikely to be of much help. For instance, fix a unit vector and let $K$ be the orthogonal projection onto its span. Now you give me an arbitrary bounded operator $T$ and I notice that the commutator with my $K$ has rank at most 2. Would that really help me to show that $T$ has a nontrivial invariant subspace? Jun 23 '20 at 14:55
• That said, even if you don't solve the ISP you might well discover something interesting in this direction ... Jun 23 '20 at 14:57
• @NikWeaver Nice point! Indeed, my idea was to try to construct a particular rank one operator $K$ which could help to show the existence of a nontrivial invariant subspace, because as you noticed the use of any $K$ would probably be not so useful Jun 23 '20 at 15:02