That equation is called a (continuous-time) algebraic Riccati equation, and there is ample literature on when they are solvable; just look for this search term. For instance, the book Algebraic Riccati equations by Lancaster and Rodman, or Numerical solution of AREs by Bini, Iannazzo, Meini.
In the generic case there is a finite number of solutions that grows exponentially with the dimension $n$: think about the scalar case, for instance, which already has 2 (possibly complex) solutions if $b \neq 0$. So usually the first point is understanding which solution you need. Typically one seeks the so-called stabilizing solution (or the antistabilizing one), i.e., the one such that $A^T+BX$ has eigenvalue in the left (resp. right) half-plane.