An obvious necessary condition is that |a|, |b| and |c| can be sides of a triangle. The two-dimensional case can be analyzed further geometrically. If the triangle inequalities are satisfied, there are vectors congruent to a and b which form a triangle with c. The condition you require is that these vectors can be obtained by rotating a and b in opposite directions, a process which does not change the bisector of the angle between a and b.

An obvious necessary condition is that |a|, |b| and |c| can be sides of a triangle. The two-dimensional case can be analyzed further geometrically. If the triangle inequalities are satisfied, there are vectors congruent to a and b which form a triangle with c. The condition you require is that these vectors can be obtained either by rotating a and b in opposite directions (if X is proper orthogonal, and in this case the bisector of the angle between a and b stays the same), or by reflection across some axis (if X is improper orthogonal).