Here is a one-dimensional analogue of Richard's triangle example, obtaining a counterexample in the set of reals. Namely, replace every integer $n$ with two numbers at fixed small distance .
For example, let $X$ have $n$ and $n+\frac14$ for every integer $n$. n\pm\epsilon$. One can suitably translate and reflect to realize homogeneity, but there is no isometry swapping $0$ \epsilon$ and $1$.1+\epsilon$.

