Here is one approach, which may efficiently show lack of solution in some cases.
Make square matrix $O$ with entries variables. We have $O'=O^{-1}$ and $O'$ is the transpose.
Mulitply by $O'$. Since $OO'=I$ we get:
$O'A=CO',O'B=DO'$.
This is systems of $2n^2$ linear equations over $K$ in $n^2$ variables.
If the linear system doesn't have solution, there is no solution to the problem.
In addition we need the constraint $\det(O)=\pm 1$, which might be intractable to compute symbolically.
If the linear system doesn't have solution, there is no solution.
If $K$ is a field, find basis for the solutions and try to substitute in the nonlinear constraint, searching for solution in it.