Is there a maximal subgroup of depth 3?

Question

Let's first define what we mean by depth of a subgroup.

Let $G$ be a finite group and $H$ a subgroup. Let $(V_i)_{i \in I}$ and $(W_j)_{j \in J}$ be the irreducible complex representations of $G$ and $H$ (up to isom.). Consider the bipartite graph $\mathcal{G}$ whose vertices are these representations, and with $d_{ij}$ edges between $V_i$ and $W_j$ if $\langle V_i\vert_H,W_j \rangle = d_{ij}$. Let $\mathcal{G}_0$ be the connected component of $\mathcal{G}$ containing the trivial representation $V_0$ of $G$. Note that $\mathcal{G}_0$ can be called the principal block of the decomposition matrix, or the principal graph. Note that $\Vert \mathcal{G}_0 \Vert^2 = |G:H|$.

Definition: The depth of $H \subset G$ is the distance between $V_0$ and a farthest vertex in $\mathcal{G}_0$.

Alternative definition (after Noah): depth is the maximum number of applications of induction $\mathrm{Ind}_H^G$ or restriction $\mathrm{Res}_H$ from $V_0$ that generate a new irreducible component (by Frobenius reciprocity).

Note that the depth of $H \subset G$ is $2$ if and only if $H$ is a normal subgroup.

The principal graph for $\{e\} \subset S_3$, where the starry vertex is $V_0$:

The principal graph of $\langle (1,2)(3,4) \rangle \subset A_4$ (depth $3$):

The principal graph for $A_4 \subset A_5$ (depth $5$):

If $H \subset G$ is a maximal subgroup of depth $2$, then it is easy to see that $|G:H|$ is a prime number.
Let $I_n$ be the set of indices of maximal subgroups of depth $n$ in the finite groups. Then $I_2 = \mathbb{P}$.
In order to see what $I_n$ looks like for $n>2$, we computed the beginning of these sets, more precisely, we computed the subsets $E_n \subset I_n$ restricted to $|G:H| \le 100$, $|G| < 10^7$ and $n \le 7$. The results are the following (see full computation and code below):

• $E_2=\{2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, \dots \}$,
• $E_3=\emptyset$,
• $E_4=\{3, 4, 5, 7, 8, 9, 10, 11, 13, 15, 16, 17, 19, 23, 25, 27, 28, 29, 31, 32, 36, 37, \dots \}$,
• $E_5=\{ 5, 6, 8, 9, 10, 11, 12, 14, 15, 17, 18, 20, 21, 24, 26, 28, 30, 32, 33, 35, 36, \dots \}$,
• $E_6=\{ 4, 6, 7, 8, 9, 10, 12, 14, 16, 18, 20, 21, 24, 25, 26, 27, 28, 30, 32, 35, 36, 38, \dots\}$,
• $E_7=\{11, 13, 25, 31, 36, 40, 45, 49, 57, 64, 81, 100\}$.

Surprisingly $E_3=\emptyset$, which leads to wonder whether $I_3 = \emptyset$ also, in other words:

Question: Is there a maximal subgroup of depth $3$?

For people interested in subfactor (planar algebra) theory, the question extends as follows:
Bonus question: Is there an irreducible maximal subfactor of depth $3$ and integral index?

---

Computation

gap> DepthListPrimitive(100,10000000);
[ [ [ 1 ] ], [ [ 2 ], 2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97 ], [ [ 3 ] ],
  [ [ 4 ], 3, 4, 5, 7, 8, 9, 10, 11, 13, 15, 16, 17, 19, 23, 25, 27, 28, 29, 31, 32, 36, 37, 41, 43, 45, 47, 49,
      53, 55, 57, 59, 60, 61, 64, 65, 66, 67, 68, 71, 73, 78, 79, 81, 83, 89, 91, 97 ], [ [ 5 ], 5, 6, 8, 9, 10, 11, 12, 14, 15, 17, 18, 20, 21, 24, 26, 28, 30, 32, 33, 35, 36, 38, 42, 44,
      48, 50, 54, 55, 56, 60, 62, 65, 66, 68, 72, 74, 78, 80, 82, 84, 90, 98, 100 ],
  [ [ 6 ], 4, 6, 7, 8, 9, 10, 12, 14, 16, 18, 20, 21, 24, 25, 26, 27, 28, 30, 32, 35, 36, 38, 42, 44, 45, 48, 49,
      50, 52, 54, 56, 60, 62, 63, 64, 65, 68, 72, 74, 77, 80, 81, 82, 84, 90, 98, 100 ], [ [ 7 ], 11, 13, 25, 31, 36, 40, 45, 49, 57, 64, 81, 100 ] ]

Code (the first function is due to Jack Schmidt, see here)

PrincipalGraph:=function(g,h)
    local mat,edges;
    mat:=MatScalarProducts(Irr(h),RestrictedClassFunctions(Irr(g),h)); #Print(mat);
    edges := Filtered( Cartesian([1..Size(mat)],-[1..Size(mat[1])]), ij -> not IsZero(mat[ij[1]][-ij[2]]));
    return edges;
end;;   

DepthPrimitive:=function(d,r)
    local P,dd,c,cc,PP,a,G,H;
    G:=PrimitiveGroup(d,r);
    H:=Stabilizer(G,1);
    dd:=0;
    P:=PrincipalGraph(G,H);
    c:=[1];
    while P<>[] do
        PP:=[];
        cc:=[];
        for a in P do
            if a[1] in c then
                Add(cc,a[2]);
            elif a[2] in c then
                Add(cc,a[1]);
            else 
                Add(PP,a);
            fi;
        od;
        c:=cc;
        P:=PP;
        dd:=dd+1;
    od;
    return dd;
end;;

DepthListPrimitive:=function(n,M)
    local d,dd,R,r,L;
    L:=[[[1]],[[2]],[[3]],[[4]],[[5]],[[6]],[[7]]];
    for d in [2..n] do
        R:=NrPrimitiveGroups(d);
        for r in [1..R] do
            if Order(PrimitiveGroup(d,r))<M then
                dd:=DepthPrimitive(d,r);
                if dd<8 then
                    if not d in L[dd] then
                        Add(L[dd],d);
                    fi;
                fi;
            fi;
        od;
    od;
    return L;
end;;

Answer 1

As Noah points out, you are looking for some (core-free) maximal subgroup $H<G$ such that $1_H^G$ has nonzero inner product with every irreducible character.

Say $G=L_2(p)$ with $p$ prime and $p \equiv 1 \bmod 8$. Then $G$ has a maximal subgroup $H \cong S_4$. Every element $h \in H$ has diagonalizable preimage in $SL_2(p)$, and the conjugacy class of $h$ in $G$ is determined by $|h|$. Note $|h| \in \{1,2,3,4\}$ and that $G$ has a unique conjugacy class of elements of each such order.

Let $h_2$, $h_3$ and $h_4$ represent, respectively, the conjugacy classes of elements of orders $2,3,4$. Using the facts above and direct calculation, you should get that for any irreducible character $\chi$ of $G$,

$$ \langle 1_H^G,\chi \rangle=\frac{1}{24}(\chi(1)+9\chi(h_2)+8\chi(h_3)+6\chi(h_4)). $$

Now, assuming $\chi$ is not the trivial character, $\chi(1)$ is at least $\frac{p+1}{2}$. On the other hand, each of $\chi(h_2)$, $\chi(h_3)$ and $\chi(h_4)$ is a sum of at most two roots of unity. Thus, for large enough $p$, you get $\langle 1_H^G,\chi \rangle>0$ for all irreducible $\chi$, and $H$ has depth three.

I found the character table for $L_2(q)$ at the web page of Jeffrey Adams,

http://www.math.umd.edu/~jda/characters/characters.pdf

You might be able to pull a similar trick in other cases - take some group $H$ and a (projective) complex irrep of $H$. This should in many cases reduce to an embedding of $H$ in some simple groups of fixed Lie type over various fields, some of these embeddings making $H$ maximal. For large enough fields, you might get depth three.

Answer 2

This isn't a full answer.

First let’s translate this into purely group theoretic language.

The vertex at depth $0$ is the trivial $G$-rep, the vertex at depth $1$ is the trivial $H$-rep, the vertices at depth $0$ or $2$ are the $G$-irreps in $\mathrm{Ind}_H^G 1$, the vertices at depth $1$ or $3$ are the $H$-irreps in $\mathrm{Res}_H\mathrm{Ind}_H^G 1$, the vertices at depth $0$, $2$, or $4$ are the $G$-irreps in $\mathrm{Ind}_H^G \mathrm{Res}_H\mathrm{Ind}_H^G 1$, etc.

Let's first ask which $H$ and $G$ irreps appear eventually after successive induction-restriction. These are exactly the reps which are trivial when restricted to $N =\cap_g gHg^{-1}$. We can replace $H \subset G$ with $H/N \subset G/N$ without changing the graph (or the subfactor). So WLOG assume $N = \{1\}$ and so $G$ is a transitive subgroup of $S_n$ with $n=|G:H|$.

So what you want is a maximal subgroup $H \subset G$ such that $H$ is non-trivial (so there are vertices at depth $3$), and every $G$-irrep occurs in $\mathrm{Ind}_H^G 1$ (so there are no vertices at depth $4$).

If the index $|G:H|$ is a prime $p$. Take $P$ a Sylow $p$-subgroup of $G$. This is just a choice of $p$-cycle in $G$. The double coset space $P\backslash G/ H$ is trivial since $P$ is a transitive subgroup of $S_p$. So $$\mathrm{Res}_H \mathrm{Ind}_P^G 1 = \mathrm{Ind}_1^H 1$$ only has one copy of the trivial. Hence any nontrivial irrep $W$ in $\mathrm{Ind}_P^G 1$ when restricted to $H$ has no trivial subrep. Hence $W$ doesn’t appear in $\mathrm{Ind}_H^G 1$.

Then $\mathrm{Ind}_P^G 1$ is a trivial representation of $G$, and so $P=G$ is a $p$-group. But $G \subset S_p$ and $p$ only divides $p!$ once, hence $G$ is cyclic of order $p$ and $H$ is trivial. It follows that $H \subset G$ is depth $2$, contradiction with depth $3$.

Conclusion: There is no maximal subgroup of depth $3$ and prime index, i.e. $I_3 \cap \mathbb{P} = \emptyset$.

Answer 3

The investigation of the finite simple groups $G$ with $|G|<10^6$ (with the code below) reveals the following maximal subgroups of depth $3$:

• $A_4 \subset L_2(q)$ for $q=27,37,43,53,67,83,107$,
• $S_4 \subset L_2(q)$ for $q= 41,71,73,79,89,97,103,113$,
• $F_7 \subset J_1$, with $F_7$ a Frobenius group, and $J_1$ a Janko group,
• $A_5 \subset L_2(q)$, for $q=101,109,125$.

The first example $(A_4 \subset L_2(27))$ has index $819$. The matrix of its (bipartite) principal graph is:
$$\left( \begin{matrix} 1 & 1 & 1 & 2 & 2 & 2 & 1 & 1 & 1 & 2 & 3 & 3 & 3 & 3 & 3 & 3 \\ 0 & 3 & 0 & 3 & 3 & 3 & 2 & 2 & 2 & 2 & 2 & 2 & 2 & 2 & 2 & 2 \\ 0 & 0 & 3 & 3 & 3 & 3 & 2 & 2 & 2 & 2 & 2 & 2 & 2 & 2 & 2 & 2 \\ 0 & 3 & 3 & 6 & 6 & 6 & 7 & 7 & 7 & 7 & 7 & 7 & 7 & 7 & 7 & 7 \end{matrix} \right)$$

Then, the following set of indices of these subgroups is a subset of $I_3$: $$\{ 819, 1435, 2109, 3311, 4180, 6201, 7455, 8103, 8585, 10270, 10791, 12529, 14685, 16275, 19012, 22763, 23821, 30058, 51039 \}$$

Question: Is $819$ the smallest index for a maximal subgroup of depth $3$?

---

Code

PrincipalGraph:=function(g,h)
    local mat,edges;
    mat:=MatScalarProducts(Irr(h),RestrictedClassFunctions(Irr(g),h)); #Print(mat);
    edges := Filtered( Cartesian([1..Size(mat)],-[1..Size(mat[1])]), ij -> not IsZero(mat[ij[1]][-ij[2]]));
    return edges;
end;;

DepthSubgroup:=function(G,H)
    local P,dd,c,cc,PP,a;
    dd:=0;
    P:=PrincipalGraph(G,H);
    c:=[1];
    while P<>[] do
        PP:=[];
        cc:=[];
        for a in P do
            if a[1] in c then
                Add(cc,a[2]);
            elif a[2] in c then
                Add(cc,a[1]);
            else 
                Add(PP,a);
            fi;
        od;
        c:=cc;
        P:=PP;
        dd:=dd+1;
    od;
    return dd;
end;;

Depth3MaxSubSimple:=function(n,m)
    local it,i,G,T,M,x,H,l;
    it:=SimpleGroupsIterator(n,m);
    l:=[];
    for i in it do
        Add(l,i);
    od;
    for G in l do
        M:=MaximalSubgroupClassReps(G);
        for H in M do
            if DepthSubgroup(G,H)=3 then
                Print([G,H,IdGroup(H),Order(G),Order(H),Order(G)/Order(H)]);
            fi;
        od;
    od;
end;;