Determining if a matrix is orthogonal Let g be an element of $GL_n(\mathbb C)$. We know that there are orthogonal groups $O(\beta)=\{X\in GL_n(\mathbb C) \mid X^t\beta X=\beta\}$ for any $\beta$, invertible symmetric matrix. Though these groups are conjugate since all symmetric bilinear forms over $\mathbb C$ are equivalent. In literature one can find a way to determine when a matrix $X$ satisfies $X^tX=I$ thus an element of the orthogonal group $O(I)$. 
My question is as follows: Is there a way to determine if an invertible matrix X belongs to $O(\beta)$ for some $\beta$. To me it seems that I have to check for all conjugates. Sure enough there will be an easier way out!  
More precisely I want to know the following: Given $X$ an invertible matrix how to determine $\beta$, an invertible symmetric matrix, so that $X^t\beta X=\beta$? 
 A: Consider the entries of $X^t \beta X - \beta$ as linear equations in the entries of the symmetric matrix $\beta$, and solve.  For $n \times n$ matrices these are 
$(n+1)n/2$ equations in $(n+1)n/2$ unknowns.   If there is a nontrivial solution, compute the rank of $\beta$ (using generic values for the free variables).
EDIT: It probably should be mentioned that this is a similarity invariant, i.e.
if $\beta$ works for $X$ then $S^t \beta S$ works for $S^{-1} X S$ where $S$ is any invertible matrix.
In the cases $n=4$ and $n=5$, I tried companion matrices of self-reciprocal polynomials $p(x) = 1 + a_1 x + a_2 x^2 + a_1 x^3 + x^4$ and $p(x) = 1 + a_1 x + a_2 x^2 + a_2 x^3 + a_1 x^4 + x^5$ respectively.   It turns out that for $n=4$ there are invertible solutions $\beta$ iff
$a_1 \ne \pm (1 + a_2/2)$, and for $n=5$ iff $a_1 \ne -1 - a_2$.
A: Interesting question !
Let us denote $L_X:\beta\mapsto X^T\beta X$. This is an endomorphism of ${\bf Sym}_n({\mathbb C})$.

Lemma. The characteristic polynomial of $L_X$ is
  $$\prod_{i\le j}(t-\mu_i\mu_j)$$
  where $\mu_1\ldots,\mu_n$ are the eigenvalues of $X$.

Sketch of the proof : the characteristic polynomial has degree $n(n+1)/2$. Suppose that $X$ is diagonalisable, and that the products $\mu_i\mu_j$ are pairwise distinct. Then each $\mu_i\mu_j$ is an eigenvalue, associated with an eigenvector $\ell^T_i\ell_j+\ell^T_j\ell_i$ where $\ell_iX=\mu_i\ell_i$. So these are all the eigenvalues and the formula is correct in this case. Because the characteristic polynomial of $L_X$ is also a polynomial in the entries of $X$, as well as the product indicated above, the equality stands in the closure (topological or Zariski) of the special case, which is ${\bf M}_n(\mathbb C)$.
Now, what are we looking for ? An eigenvector $\beta$, non-singular and associated with the eigenvalue $1$. Thus we need $\mu_i\mu_j=1$ for some pair. But not only for some, because $\ell^T_i\ell_j+\ell^T_j\ell_i$ is singular (unless $n=2$). What we really need is enough multiplicity that we can pick a non-singular eigen-matrix. If $X$ is diagonalizable, this means that the eigenvalues be associated pairwise:
$$\mu_1\mu_n=\mu_2\mu_{n-2}=\cdots=1.$$
Notice that if $n$ is even, this yields the necessary condition that $\det X=1$. It was obvious a priori that $(\det X)^2=1$, but actually the case $-1$ is excluded. This ressembles the well-known fact that symplectic matrices have determinant $+1$.
Edit. A more direct and definitive way to arrive to this conclusion : if $\beta$ exists, then $X^{-T}=\beta X\beta^{-1}$, that is $X^T$ is conjugated to $X^{-1}$. This proves that $\lambda\mapsto\lambda^{-1}$ is an involution of the spectrum of $X$. Conversely, suppose that $X^t$ and $X^{-1}$ are conjugated ; can we choose a symmetric conjugation matrix ? The answer is no, as shown by the counterexample
$$X=\begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}.$$
A: There is a complete characterization of matrices that belong to at least one
orthogonal group. It reads as follows over any arbitrary field $\mathbb{F}$
with characteristic different from $2$ (with algebraic closure denoted by $\overline{\mathbb{F}}$:

Given a matrix $M \in \mathrm{GL}_n(\mathbb{F})$, there exists an
  invertible symmetrix matrix $\beta$ such that $M^T \beta M=\beta$ if
  and only if, for every $\lambda \in \overline{\mathbb{F}} \setminus \{0,1,-1\}$ and 
  every positive integer $k$, one has
  $\mathrm{rk}(M-\lambda I_n)^k=\mathrm{rk} (M-\lambda^{-1} I_n)^k$ and,
  for each one of the (possibly absent) eigenvalues $1$ and $-1$ and
  every positive integer $k$, there is an even number of Jordan cells of
  size $2k$ in the Jordan reduction of $M$. 

Moreoever, if you have access to the Jordan reduction of $M$ and the above conditions
are satisfied, then coming up with an explicit solution $\beta$ is not difficult.
This characterization has been known for a very long time. See my preprint http://arxiv.org/abs/1008.4458 for a recent account using elementary methods.  
A: Not quite an answer: If you replace "symmetric" by hermitian, then, and transpose by conjugate transpose, then, since every hermitian matrix can be written by as $\beta = A^* A,$ then $X^* \beta X = \beta$ implies that a conjugate of $X$ lies in the unitary group, which is necessary and sufficient. That is the same as saying that the eigenvalues of $X$ are of modulus one (and the matrix is diagonalizable).The same works with "symmetric" and "transpose" but over $\mathbb{R}.$ 
ADDITION for symmetric matrices, there is a canonical form (due to Horn and Sergeichuk: Canonical forms for complex matrix congruence
and ∗congruence
Roger A. Horn a,∗
, Vladimir V. Sergeichuk b [Linear Algebra and Applications, 2006]. ) See, equations (7) and on in the paper. There are three types of blocks, and clearly a matrix is in the orthogonal group of some complex symmetric matrix if it is conjugate to a direct sum of matrices in the orthogonal groups of the three block types.
