I read the reference of Igor, but I have nothing shot about our equation !

There is a case that seems less difficult. If $A,B$ are symmetric complex, then the problem is essentially equivalent to solve $2$ equations of the form $ZZ^T=D$, that is, the case $A=I$ or $B=I$.

equation 1: $YY^T=A$ , equation 2: $ZZ^T=B$ , equation 3: $XY=Z$. If $A,B$ are invertible, then $Y,Z$ exist and $X=ZY^{-1}$.In these conditions, can we find, at least in theory, all solutions ?

For Halder. The equation $XAX=B$ is easy to solve (at least when $A$ is invertible) because it can be rewritten $(XA)^2=BA$.

I read the reference of Igor, but I have nothing shot about our equation !

There is a case that seems less difficult. If $A,B$ are symmetric complex, then the problem is essentially equivalent to solve $2$ equations of the form $ZZ^T=D$, that is, the case $A=I$ or $B=I$.

equation 1: $YY^T=A$ , equation 2: $ZZ^T=B$ , equation 3: $XY=Z$. If $A,B$ are invertible, then $Y,Z$ exist and $X=ZY^{-1}$.In these conditions, can we find, at least in theory, all solutions ?

For Halder. The equation $XAX=B$ is easy to solve (at least when $A$ is invertible) because it can be rewritten $(XA)^2=BA$.

EDIT: Let $A,B\in\mathcal{M}_n(\mathbb{C})$. According to a result by Turnbull, Aitken (cited in the Igor's reference) our equation admits a solution iff $A,B$ are congruent iff there are invertible $P,Q$ s.t. $PAQ=B,PA^TQ=B^T$. Then, for generic $A,B$, the equation $XAX^T=B$ has no solutions. Thus, the "good" equation is $XAX^T=CAC^T$. Using my computer, I randomly choose real matrices $A,C$ ; I find that the algebraic set of complex solutions has dimension $1$ when $n=2$, $1$ when $n=3$, $2$ when $n=4$, $2$ when $n=5$.