Let $N \geq 5$ be a prime, and $H$ a subgroup of $GL_2(\mathbb{F}_N)$. As shown in Chapter IV of [DeRap], there is a curve $X_H(N)$, defined over $K_N := \mathbb{Q}(\zeta_N)^{\det H}$, which is a coarse moduli space of elliptic curves with level-$H$ structure; that is, given any $L$ an extension of $K_N$, the $L$-points of $X_H(N)$ correspond to $\bar{L}$-isomorphism classes of elliptic curves over $L$ whose associated mod-$N$ representation on $N$-torsion points is contained in (a conjugate of) $H$. (Sometimes this is a fine moduli space, but I don't think this is important for this question).
It is certainly not the case that, if $E/L$ has level-$H$ structure at $N$, then so too do all twists of $E$. For example, if $H$ were the subgroup $\left\{\left(\begin{array}{cc}1 & \ast \\ 0 & \ast \end{array}\right)\right\}$, corresponding to $X_1(N)$, then it is not true.
But this is true sometimes; e.g., if $H = \left\{\left(\begin{array}{cc}\ast & \ast \\ 0 & \ast \end{array}\right)\right\}$, the Borel subgroup; this follows from the slogan: if an elliptic curve has a rational cyclic isogeny, then so too do all twists. A similar argument sows this for $H$ being the normaliser of a split Cartan subgroup, viz $\left\{\left(\begin{array}{cc}\ast & 0 \\ 0 & \ast \end{array}\right)\right\} \bigcup \left\{\left(\begin{array}{cc}0 & \ast \\ \ast & 0 \end{array}\right)\right\}$. I'll say that these level structures are twist-invariant.
This leads me to ask:
Is there a characterisation of the twist-invariant level structures? What is it about a subgroup $H$ that makes it twist-invariant?
[DeRap] : P. Deligne, M. Rapoport, ``Les schemas de modules de courbes elliptiques''.
