Can two unitary similar real matrices be orthogonal similar? suppose $A=U^tBU$ where $U$ is unitary, does there always exists a real orthogonal matrix $O$, such that $A=O^tBO$ ?

$\begingroup$ I presume with $U^t$ you mean $U^{1}=U^{\ast}$ (so the conjugate transpose rather than just the transpose) $\endgroup$– Carlo BeenakkerAug 31 '13 at 8:48

$\begingroup$ "Can" in the title and body are apparently meant to be "Must" (the answer to the "can" question is trivially affirmative taking $A=B$). Also I don't think this is really a research level question, and it would be better suited at math.stackexchange.com $\endgroup$– Marc van LeeuwenAug 31 '13 at 9:03
Unitarily similar real matrices are always orthogonally similar. The proof can be found in Rached Mneimné and Frédéric Testard's book "Introduction à la théorie des groupes de Lie classiques".
First of all, one sees that two complex square matrices $A$ and $B$ are unitarily similar if and only if the pairs $(A,A^\star)$ and $(B,B^\star)$ are similar over the complex numbers. The direct implication is straightforward; for the converse, assume that we have an invertible matrix $P$ such that $A=PBP^{1}$ and $A^\star=P B^\star P^{1}$. Applying $()^\star$ to the second equality, we get $A=(P^\star)^{1}BP^\star$, and using the first equality we deduce that $P^\star P$ commutes with $B$. Writing the polar decomposition of $P$ as $P=UH$, this means that $H^2$ commutes with $B$, which, as $H$ is a polynomial of $H^2$, yields that $H$ commutes with $B$. Therefore $UBU^{1}=PBP^{1}=A$, QED.
Now, with the same line of reasoning, one shows that two real square matrices $A$ and $B$ are orthogonally similar if and only if the pairs $(A,A^t)$ and $(B,B^t)$ are similar over the real numbers.
One concludes by noting that if two pairs $(A_1,A_2)$ and $(B_1,B_2)$ of real square matrices are similar over the complex numbers, then they are also similar over the real numbers: this is a widely known special case of the NoetherDeuring Theorem (for a classical elementary proof, see section 2.2. of Invariance of simultaneous similarity and equivalence of matrices under extension of the ground field).