As a beginning of a search for solutions, we can take norms of each side.  We then get 

$$||a||^2 + ||b||^2 + 2 b\cdot (X^2 a)=||c||^2,$$

which is enough to fix the angle between $b$ and $X^2 a$.

Similarly, multiplying through by $X$ and then dotting with $b$, we can conclude that

$$b\cdot (Xc)=||b||^2+b\cdot (X^2 a)=||b||^2+\frac{||c||^2-||a||^2-||b||^2}{2}=\frac{||b||^2+||c||^2-||a||^2}{2}.$$

Therefore, we know the angle that $b$ makes with $Xc$.  Similar calculations show us the angle that $a$ makes with $X^T c$.  In two dimensions, this is enough to find $X$ geometrically if it exists in most cases, and otherwise to say that there is no such solution.    The geometry is slightly more involved in 3 dimensions, and I'm not immediately sure if there is more useful information to be extracted through dot products to help.