As a beginning of a search for solutions, we can take norms of each side. We then get
$$\|a\|^2 + \|b\|^2 + 2 b\cdot (X^2 a)=\|c\|^2,$$
which is enough to fix the angle between $b$ and $X^2 a$.
Similarly, multiplying through by $X$ and then dotting with $b$, we can conclude that
$$b\cdot (Xc)=\|b\|^2+b\cdot (X^2 a)=\|b\|^2+\frac{\|c\|^2-\|a\|^2-\|b\|^2}{2}=\frac{\|b\|^2+\|c\|^2-\|a\|^2}{2}.$$
Therefore, we know the angle that $b$ makes with $Xc$. Similar calculations show us the angle that $a$ makes with $X^T c$. In two dimensions, this is enough to find $X$ geometrically if it exists in most cases, and otherwise to say that there is no such solution. The geometry is slightly more involved in 3 dimensions, and I'm not immediately sure if there is more useful information to be extracted through dot products to help.
Here is an approach for $\mathbb R^3$, inspired by Michael Renardy's answer.
Let us temporarily expand the problem to $Aa+Bb=c$, where $A,B\in O_n(\mathbb R)$. Assuming that $|a|,|b|$, and $|c|$ satisfy the triangle inequality, we can find a solution, $(A_0,B_0)$. However, the space of all solutions is $(MA_0,MB_0)$ where $Mc=c$ and $M\in O_n(\mathbb R)$. Thus, we've reduced the problem to:
Given $A,B\in O_n(\mathbb R)$ and $c\in \mathbb R^n$, does there exist $X\in O_n(\mathbb R)$ such that $$Xc=c \quad \text{and} \quad I=AXBX.$$
Since $AX$ and $BX$ are inverses, they commute, and so $BXAX=I$ too. Evaluating at $c$, we get 3 equations, $$ Xc=c, \quad X(Bc)=A^Tc, \quad X(Ac)=B^Tc.$$ Assuming that $c, Ac, Bc$ span your space (which happens in $\mathbb R^3$ for most $(A,B,c)$-triples), this specifies a unique candidate $X$ to test to see if it actually satisfies the problem.
Explicitly, if $P$ is the matrix whose columns are $c,Ac, Bc$ respectively, and $Q$ is the matrix whose columns are $c, B^Tc, A^Tc$ respectively, then $X=QP^{-1}$. We just need to check that $XX^T=I$ and $AXBX=I$. or see that these equations are violated.
