Suppose I have $A\in U(n)$ such that $A^t=A$ (which is a bit un-natural, as usually you'd consider the hermitian transpose, not the transpose).

Well, then $A=X+iY$ say, for $X$ and $Y$ real matrices. Thus $X$ and $Y$ are both symmetric, and also $I = A^*A=(X-iY)(X+iY)$ so $X^2+Y^2=I$ and $XY=YX$. So I can find an orthogonal matrix $V$ with $V^t X V$ and $V^t Y V$ both diagonal. We conclude that $V^t A V$ is diagonal, with diagonal entries from $\mathbb T$. That is, $V^tAV$ is a diagonal unitary matrix.

Suppose now I look at indefinite situation. So I let $J$ be a diagonal matrix consisting of $n$ 1s and $m$ -1 entries, say. Then $A\in U(n,m)$ if and only if $A^*JA=J$, and similarly $A\in O(n,m)$ if and only if $A^tJA=J$.

Suppose I now have $A\in U(n,m)$ with $A^t = JAJ$. Is it possible to conjugate $A$ by some $V\in O(n,m)$ into a "nice" form?

What's a good reference for this sort of stuff?

Matrix Analysisof HJ is titled: "Complex symmetric matrices"--this section has several interesting things worth looking at! – Suvrit Nov 25 '10 at 9:29