The golden rule for conjectures in operator theory:

Every ad-hoc conjecture is most likely false for $2 \times 2$-matrices. :-)

So here's a $2 \times 2$-counterexample for the question: Let $A = \mathbb{C}^{2 \times 2}$ and $$ x = \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix} \quad \text{and} \quad y = \begin{pmatrix} 4 & 3 \\ 3 & -2 \end{pmatrix} $$.