I tried the $n=2$ case, using Maple to find a "plex" Groebner basis for the equations. The result is rather complicated, and too large to show here. The first member of the basis, for example, is $y_1$ times an irreducible polynomial of $143$ terms and total degree $8$ in the entries of $A$ and $B$, which must be $0$ in order to have a solution with $y_1 \ne 0$. So I don't think you'll get a "simple" criterion.
Robert Israel
- 54.2k
- 1
- 76
- 152