Skip to main content
2 of 2
added 35 characters in body
Ian Morris
  • 6.2k
  • 2
  • 31
  • 64

To find such an operator in separable real Hilbert space just take a countable direct sum of rotations of $\mathbb{R}^2$. More formally:

Let $(e_n)_{n \in \mathbb{N}}$ be a countable basis for Hilbert space and define an operator $A$ by $Ae_{2n}=e_{2n+1}$ and $Ae_{2n+1}=-e_{2n}$. If $u=\sum_{k=1}^\infty a_ke_k$ where the $a_k$'s are real then we have $Au=\sum_{n=1}^\infty (a_{2n}e_{2n+1}-a_{2n+1}e_{2n})$ and so $\langle Au, u\rangle = \sum_{n=1}^\infty (a_{2n+1}a_{2n}-a_{2n}a_{2n+1})=0$. If the coefficients are allowed to be complex then the minus becomes a plus in the imaginary component and we will typically get a nonzero answer due to the effect of complex conjugation in the definition of inner product.

As I mentioned in the comments, for real matrices of odd finite dimension the characteristic polynomial of $A$ will have a real root, which implies the existence of an invariant subspace. If the corresponding eigenvalue is nonzero then the relation $\langle Au, u\rangle=0$ is obviously impossible when $u$ belongs to this eigenspace, so in odd dimensions this property is impossible for an invertible matrix $A$. This same observation means that the desired property also does not hold for any invertible complex matrix.

Ian Morris
  • 6.2k
  • 2
  • 31
  • 64