Automorphism groups of elliptic curves are very well understood. Of course, every elliptic curve has the automorphism $[-1]$ of order $2$. If we are over a (algebraically closed) field, this is the only non-trivial automorphism iff the $j$-invariant of our elliptic curve is neither $0$ nor $1728$. Over $\mathbb{C}$ the only other possibilities for automorphism groups are $\mathbb{Z}/4$ and $\mathbb{Z}/6$ occuring for $E = \mathbb{C}/\Lambda$ where $\Lambda$ is the square lattice or the "honeycomb" lattice respectively. In particular, the generic elliptic curve has automorphism group $\mathbb{Z}/2$, i.e. the locus of elliptic curves with bigger automorphism group in the moduli space of elliptic curves has codimension bigger than $0$.

I am interested in the analogous question for abelian varieties. To force all the automorphism groups to be finite, it seems sensible to consider (principally) polarized abelian varieties.

Let $\mathcal{A}_g$ be the moduli space of (principally) polarized complex abelian varieties of dimension $g$. What is the codimension of the locus of such varieties with automorphism group bigger than $\mathbb{Z}/2$ ?

Likewise, one can ask the same question not about arbitrary principally polarized abelian varieties, but such equipped with an endomorphism and level structure. So, in addition to the polarization on our (complex) abelian variety $A$ we also want a ring homomorphism $\mathcal{O}_F \to End(A)$ (for a fixed, say, quadratic imaginary field $F$), compatible with the Rosatti involution, and a level structure (plus some determinant/trace-condition). For a generic such $A$, its automorphism group may contain now $\mathcal{O}_F^\times$, depending on the level structure.

Let $Sh_g$ be the moduli space of such complex abelian varieties with PEL-structure (with the choices of $F$ and level implicit) of dimension $g$. Let $G$ be the smallest automorphism group of a point in $Sh_g$. What is the codimension of the locus of points of $Sh_g$ having automorphism group bigger than $G$?