Let $X$ be a complete Riemannian space. Let us denote by $Iso(X)$ the group of isometries of $X$. It is a well-known fact that the group $Iso(X)$, when endowed with the compact-open topology, is a Lie group. Let $X\times X$ be the cartesian prodcut of $X$ with itself endowed with the product metric. Let $G:=(Iso(X)\times Iso(X))\rtimes S_2$, where $S_2$ is the symmetric group of order $2$ which permutes the two coordinates. We have a natural inclusion of $G\leq Iso(X\times X)$.
Q: Is there a general criterion on $X$ which allows us to determine when is $G=Iso(X\times X)$ ?