I would like to solve the following problem:

$$\begin{array}{ll} \text{minimize} & \mathbf{x}^T \mathbf{A} \mathbf{x}\\ \text{subject to} & \mathbf{x}^T\mathbf{B}\mathbf{x} = 0\\ & \mathbf{x}^T \mathbf{x} = 1\end{array}$$

where $\bf x$ is a vector, $\bf A, \bf B$ are square matrices, and $\bf A$ is symmetric.

Here is my thinking:

Use the Lagrange multiplier method, \begin{equation} \mathcal L (\bf x, \lambda, \mu) = \mathbf{x}^T \mathbf{A} \mathbf{x} - \lambda \mathbf{x}^T\mathbf{x} - \mu \mathbf{x}^T \mathbf{B} \mathbf{x}. \end{equation} Take the derivative with respect to $\bf x$, we get: \begin{equation} \bf{A x = \lambda x + \mu Bx} \end{equation} This is not exactly an eigenvalue problem or a generalized one. What's next?

I can apply the constraints and get $\lambda = \bf x^TAx$, $\mu = \bf x^TB^TAx/(x^TB^TBx)$. But I am looking for a method that can turn the problem to a linear problem, e.g. generalized eigenvalue problem, so that I can apply the standard numerical linear algorithms. In principle, if I can solve $\det (A-\lambda I - \mu B) = 0$, I can eliminate, say, $\mu$. But this is not feasible, numerically. A perturbative solution with $|\mu|\ll 1$ is acceptable.

*Question: Are there any methods, ideally using standard numerical linear algorithm, to solve this problem?*

These problems are similar but not the same:

Linearly constrained eigenvalue problem

Thank you in advance.

**Edit**: In viewing of the comments, I removed the "full rank" condition and does not requires $\bf A$ to be "positively defined". Hopefully, the problem may have a solution?

The background of the problem is as follows: $\bf A$ is a Hamiltonian. $\bf x$ is its eigenvector with lowest energy. $\bf x^T Bx = 0$ represents a constraint imposed by a symmetry. In practice, $\bf A$ is truncated, and $\bf x^T B x \ne 0$.

Now, I am trying to reformulate the problem to guarantee the symmetry constraint $\bf x^T B x = 0$. As a result, $\bf x$ may not be an eigenvector of $\bf A$, which is the price to pay. My hope is that as the symmetry violation is small enough, the problem may still have an efficient solution. Hope this helps.