Group action on R^n I am studying the symmetries of a particular function,
$$
f: R^n \rightarrow R
$$
which leave $f(x)$ unchanged (i.e. so $f(Ax) = f(x)$ for some matrix $A \in R^{n \times n}$). I have found that my function is invariant under the action of a matrix group which satisfies the following equation:
$$
A X A^T = X
$$
where $A$ is an element of the group I am trying to find, and $X = x x^T$ for some $x \in R^n$.
I am not having much luck in finding the group that $A$ belongs to. Certainly $A = \pm I$ is a solution to the above equation, and I have managed to convince myself using Mathematica that for the $n = 2$ case there do exist nontrivial solutions for $A$. Does anyone have experience with this type of equation, and could you point me in the direction of where I might find out more about the properties of this particular matrix group?
EDIT: Someone asked for the explicit function so here it is in case it helps:
$$
f(x) = \left|\left|
\left(
\begin{array}{c}
x^T w_1 \\
x^T w_2 \\
x^T w_3
\end{array}
\right)
\right| \right| =
\sqrt{w_1^T x x^T w_1 + w_2^T x x^T w_2 + w_3^T x x^T w_3}
$$
Here, $w_1,w_2,w_3$ are fixed vectors in $R^n$, and you can see from the above expression that $f(Ax) = f(x)$ leads to the equation above ($AXA^T = X$). Though its possible there could be a more general condition on $A$ which satisfies
$$
\left|\left|
\left(
\begin{array}{c}
x^T w_1 \\
x^T w_2 \\
x^T w_3
\end{array}
\right)
\right| \right| =
\left|\left|
\left(
\begin{array}{c}
x^T A^T w_1 \\
x^T A^T w_2 \\
x^T A^T w_3
\end{array}
\right)
\right| \right|
$$
 A: This is a very degenerate version of an orthogonal group, due to the fact that your $X$ is not of full rank. It is rank one (assuming your $x$ is not $0$), so the quadratic form given by $X$ has signature $(1,0,n-1)$, where the last $n-1$ refers to the dimension of the subspace $K$ consisting of $y$ such that $\langle y,z\rangle=0$ for all $z\in \mathbb R^n$. This degenerate orthogonal group is known (in a geometric algebra context) to be a Levi component of a parabolic subgroup of a larger, non-degenerate orthogonal group. Either with that prior knowledge, or by direction computation, your group is a semi-direct product of $O(1)\times GL(n-1,\mathbb R)$ with $\mathbb R^{n-1}$, with the natural action of $O(1)\times GL(n-1)$ on $\mathbb R^{n-1}$.
A: If I understand correctly, you would like to determine the group $G$ of all $A\in GL(n,\mathbb R)$ with $f(Ax)=f(x)$ for all $x\in\mathbb R^n$ where $f$ is of a very particular form. You noticed that if $Axx^TA^T=xx^T$ for all $x\in\mathbb R^n$ then $A\in G$. That is undoubtedly true but doesn't help much since $A=\pm I$ is the only solution. The condition means that $A$ fixes all rank-1-quadratic forms therefore all forms which leaves only $\pm I$.
We have
$$
f(x)^2=(w_1^Tx)^2+(w_2^Tx)^2+(w_3^Tx)^2.
$$
Now $G$ depends on the choice of the vectors $w_1,w_2,w_3$. For example, if $w_1=w_2=w_3=0$ then $f=0$ and $G=GL(n,\mathbb R)$. Let me treat the opposite case: $w_1, w_2,w_3$ are linearly independent. Then, after a coordinate change one may assume that $w_1=e_1$, $w_2=e_2$, $w_3=e_3$ where the $e_i$ are the canonical basis vectors. Then $f^2$ is simply
$$
f(x)^2=x_1^2+x_2^2+x_3^2.
$$
A direct computation (or a reasoning as in Paul Garret's answer) shows that $A\in G$ means that $A$ has the block form
$$
\left(\matrix{A_{11}&A_{12}\cr A_{21}&A_{22}\cr}\right)
$$
where $A_{11}\in O(3)$ and $A_{12}=0$. So $G=[O(3)\times GL(n-3,\mathbb R)]\ltimes \text{Mat}(n-3\times 3,\mathbb R)$.
I leave the examination of the other cases for the $w_i$ as an exercise.
A: I am a little confused by Paul's answer. Your relation can be written as
$Ax x^\perp A^\perp = x x^\perp,$ which is the same as saying that 
$A$ is norm preserving, hence orthogonal.
