Let $A \in GL_n(\mathbb R)$ be fixed. Let us consider the conjugation action by $G \in GL_n(\mathbb R)$, i.e., $G^{-1}AG$. I would like to see a way to identify the matrices such that the action fixes the first column of $A$. That is, what can we say about the set
\begin{align*}
\mathcal E = \{G \in GL_n(\mathbb R): G^{-1}AGe_1 = a_1\},
\end{align*}
where $e_1$ is the standard basis vector and $a_1$ is the first column of $A$. In particular, is the set connected? Let us **exclude** the trivial case: $A$ is a multiple of $I$ or $-I$.

If we define a linear map $\phi: M_n(\mathbb R) \to \mathbb R^n$ by $X \mapsto (XA-AX)e_1$, then $\mathcal E = \ker(\phi) \cap GL_n(\mathbb R)$.