Product of commuting nonnegative operators

Question

Let $V$ be a real vector space with an inner product and $A,B : V \to V$ linear maps which are self-adjoint nonnegative-definite, i.e. $\langle Ax,y \rangle = \langle x,Ay \rangle$ and $\langle Ax,x \rangle \geq 0$, likewise for $B$. Assume that $A$ and $B$ commute (but not that $A,B$ are bounded). Can we conclude that then also $AB$ is nonnegative-definite?

This is true when $V$ is complete, because then $A$ is bounded (Hellinger-Toeplitz theorem), hence has a square root, and the claim follows easily from $A \, B = \sqrt{A} \, B \, \sqrt{A}$. It is also true when $A$ (or $B$) is essentially selfadjoint, i.e. when its closure in the completion of $V$ is self-adjoint (Theorem 3.1. in Sebestyen, Stochel, On products of unbounded operators, Acta Math. Hungar. 100(2003), 105-129).

What about the general case?

Answer 1

The answer is no. Let me sketch the proof.

Statement 1: Let $\sum^2\subseteq\mathbb R[x,y]$ denote the set of finite sums of squares of polynomials. Then the set $C=\sum^2+x\cdot\sum^2+y\cdot\sum^2$ is a convex cone in $\mathbb R[x,y].$ (Clear)

Statement 2: $C$ is closed in the finest locally-convex topology. (The proof is non-trivial. See e.g. Schmüdgens book "Unbounded Operator Algebras ..." for the proof of similar statements).

Statement 3: $xy\notin C.$ (Easy proof by contradiction).

Statement 4: There exists a linear functional $\varphi:\mathbb R[x,y]\to\mathbb R$ such that $\varphi(xy)<0$ and $\varphi(C)\geq 0.$ (Hahn-Banach separation).

Statement 5: Let $\varphi:\mathbb R[x,y]\to\mathbb R$ be a linear functional such that $\varphi(\sum^2)\geq 0$. Then there exists a GNS-representation of $\varphi.$ That is, there exists a real inner-product space $V,$ a homomorphism $\pi:\mathbb R[x,y]\to L(V)$ such that every $\pi(f),\ f\in\mathbb R[x,y]$ is symmetric, and a vector $\xi\in V$ such that $\varphi(\cdot)=\langle\pi(\cdot)\xi,\xi\rangle.$ Thereby $\xi$ is cyclic, that is $\{\pi(f)\xi,\ f\in\mathbb R[x,y]\}=V.$ (See Schmüdgens book "Unbounded Operator Algebras" Chapter 8.6)

Proof. Let $\varphi$ be as in Statement 4 and $(V,\pi,\xi)$ be as in Statement 5. Since $\xi$ is cyclic, for every $v\in V$ there exists $f_v\in \mathbb R[x,y]$ such that $v=\pi(f_v)\xi.$ Hence $$\langle\pi(x)v,v\rangle=\langle\pi(x)\pi(f_v)\xi,\pi(f_v)\xi\rangle=\langle\pi(xf_v^2)\xi,\xi\rangle=\varphi(xf_v^2)\geq 0,\ \langle\pi(y)v,v\rangle\geq 0,$$ i.e. $A=\pi(x)$ and $B=\pi(y)$ are nonnegative. On the other hand $\langle\pi(xy)\xi,\xi\rangle=\varphi(xy)<0.$ That is, $AB=\pi(x)\pi(y)=\pi(xy)$ is not nonnegative.

I believe there exists a constructive solution, see my question.