As a beginning of a search for solutions, we can take norms of each side.  We then get 

$$||a||^2 + ||b||^2 + 2 b\cdot (X^2 a)=||c||^2,$$

which is enough to fix the angle between $b$ and $X^2 a$.

Similarly, multiplying through by $X$ and then dotting with $b$, we can conclude that

$$b\cdot (Xc)=||b||^2+b\cdot (X^2 a)=||b||^2+\frac{||c||^2-||a||^2-||b||^2}{2}=\frac{||b||^2+||c||^2-||a||^2}{2}.$$

Therefore, we know the angle that $b$ makes with $Xc$.  Similar calculations show us the angle that $a$ makes with $X^T c$.  In two dimensions, this is enough to find $X$ geometrically if it exists in most cases, and otherwise to say that there is no such solution.    The geometry is slightly more involved in 3 dimensions, and I'm not immediately sure if there is more useful information to be extracted through dot products to help.

-------------------

Here is an approach for $\mathbb R^3$, inspired by Michael Renardy's answer.  

Let us temporarily expand the problem to $Aa+Bb=c$, where $A,B\in O_n(\mathbb R)$.  Assuming that $|a|,|b|$, and $|c|$ satisfy the triangle inequality, we can find a solution, $(A_0,B_0)$.  However, the space of all solutions is $(MA_0,MB_0)$ where $Mc=c$ and $M\in O_n(\mathbb R)$.  Thus, we've reduced the problem to:

Given $A,B\in O_n(\mathbb R)$ and $c\in \mathbb R^n$, does there exist $X\in O_n(\mathbb R)$ such that $$Xc=c \quad \text{and} \quad $I=AXBX.$$ 

Since $AX$ and $BX$ are inverses, they commute, and so $BXAX=I$ too.  Evaluating at $c$, we get 3 equations, 
$$ Xc=c, \quad X(Bc)=A^Tc, \quad X(Ac)=B^Tc.$$
Assuming that $c, Ac, Bc$ span your space (which happens in $\mathbb R^3$ for most $(A,B,c)$-triples), this specifies a unique candidate $X$ to test to see if it actually satisfies the problem.

Explicitly, if $P$ is the matrix whose columns are $c,Ac, Bc$ respectively, and $Q$ is the matrix whose columns are $c, B^Tc, A^Tc$ respectively, then $X=QP^{-1}$.  We just need to check that $XX^T=I$ and $AXBX=I$. or see that these equations are violated.