5 small correction

4 Jan 2011: Edited to fix discussion of verbal subgroups

I think $F_2$ is expected to behave differently from higher free groups. For a finite simple group $G$, I think it's expected (known?) that all epimorphisms $F_n \rightarrow G$ are equivalent up to $Aut(F_n)$ when $n > 2$, but there are many orbits for $n = 2$: in particular, the isomorphism class (orbit under the automorphism group of $G$) of the image of the commutator of the generators is an invariant.

Epimorphisms $F_n \rightarrow G$ are equivlaent to $n$-tuples of generators of elements of $G$ that generate. An $n$-tuple that does not generate $G$ must be contained in a proper subgroup; for a finite simple group, there are not many maximal subgroups of small index, so the vast majority of $n$-tuples actually generate. If there's only one orbit, the orbit size is rougly $|G|^n/|Aut(G)|$; for nonabelian finite simple groups, the outer automorphism group is small, so the orbit size is $|G|^{(n-1)}$ up to a small constant.

The corresponding characteristic subgroup is the kernel of the homomorphism $F_n \rightarrow \prod G_i$, where the product is over all epimorphisms to $G$ in the orbit, up to automorphism of $G$. A subgroup of the product of two groups is the kernel of the map of the product to a common quotient; using this, since the factors are simple, this homomorphism is an epimorphism. Therefore, the characteristic subgroup has huge index, $|G|^{|G|^{(n-1)/a}}$ for some not large $a$ (about 2 or 3).

Even for $n = 2$, the orbits are fairly large, so the corresponding characteristic subgroup have huge index.

I.e. characteristic subgroups with nonabelian simple composition factors are very sparse. Characteristic subgroups with polycyclic factor group are the vast majority, for several reasons: there are a lot of polycyclic groups, they tend to have large automorphism groups, and different polycyclic quotients of a free group tend to correlate, so the homomorphism to a product of a number of polycyclic quotients is often far from surjective. Thus, these characteristic subgroups do not have such a huge index. I suspect that in fact characteristic subgroups with abelian quotient should dominate. If so, the answer is boring, since such subgroups correspond 1-1 with characteristic subgroups of $Z^n$. This may be why there's not much literature on it.

Verbal Subgroups and Fully Invariant Subgroups

A word $w$ in letters $X_1, \dots, X_k$ defines a function $f_w: G^k \rightarrow G$, when $k$-tuples of elements of $G$ are substituted for the $X_i$. As explained in comments, a verbal subgroup $H_V \subset G$ is one that is generated by the images of $f_w$, for $w$ in some set $V$. In other words, $G/H_V$ is what you get by the identities for $w \in V$ that $\forall (X_1, \dots X_k)\; \; w = 1$ (as well as the relations of $G$ if $G$ is not free).

A fully invariant subgroup $H$ of a group $G$ is one such that every endomorphism of $G$ takes $H$ to a subgroup of $H$. Fully invariant subgroups are characteristic, but not necessarily vice versa.

In comments, Andy Putnam recalled that fully invariant subgroups of a free group are the ones that are supposed to be verbal, and he was dubious that characteristic subgroups of free groups are verbal. Here is an example to show he is correct: that characteristic subgroups of $F_2$ are not necessarily verbal:

We'll start from two epimorphisms $f_1$ from $F_2$ to $A_5$ that are in different orbits of the automorphism group of $F_2$: let $f1$ send the generators go to the 5-cycle $a = (1,2,3,4,5)$ and the order 2 element $b = (1,2)(3,4)$. The commutator of $a$ and $b$ is a 5-cycle. Let $f_2$ send the generators to $a$ and to the 3-cycle $c = (3,4,5)$. The commutator of $a$ and $c$ is a 3-cycle. The conjugacy class of the commutator of generators is invariant under automorphisms of $F_2$, so the maximal characteristic subgroups $H_1$ and $H_2$ containing the kernels of $f_1$ and $f_2$ are different.

Now consider any word $w(X_1, X_2, \dots, X_N)$ whose image is contained in $H_1$. Equivalently, in $F_2/H_1$, this word evaluates identically to $1$. This happens if and only if the projection to each $A_5$ factor evaluates identically to 1. Therefore, it also evaluates identically to $1$ in $F_2/H_2$. Any verbal subgroup contained in $H_1$ is in fact contained in the intersection $H_1 \cap H_2$.

To get the actual maximal verbal subgroup contained in the kernel of an epimorphism such as $f_1$, first, choose a representative for each orbit of the action of $Aut(A_5)$ on the product of copies of $A_5$ for each of the $60^2$ homomorphisms $F_2 \rightarrow A_5$, form the product of these homomorphisms, and let $P$ be the image. $P$ comes equipped with a pair of generators; let $R$ be a set of relators. Then $R$ is a verbal description for the kernel of this map.

4 Fixed discussion of verbal subgroups

4 Jan 2011: Edited to fix discussion of verbal subgroups

Verbal subgroupsSubgroups and Fully Invariant Subgroups

Here's how

A fully invariant subgroup $H$ of a group $G$ is one such that every endomorphismof $G$ takes $H$ to give a subgroup of $H$. Fully invariant subgroups are characteristic, but not necessarily vice versa.

In comments, Andy Putnam recalled that fully invariant subgroups of a free group are the ones that are supposed to be verbaldescription, and he was dubious that characteristic subgroups of free groups are verbal. Here is an example to show he is correct:that characteristic subgroups of $F_2$ are not necessarily verbal:

We'll start from two epimorphisms $f_1$ from $F_2$ to $A_5$ that are in different orbits of the characterisitic subgroup associated with automorphism group of $F_2$: let $f1$ send the generators go to the 5-cycle$a simple quotient = (1,2,3,4,5)$ and the order 2 element $Q$ b = (1,2)(3,4)$. The commutator of$F_n$:a$ and $b$ is a 5-cycle.Let $f_2$ send the generators to $a$ and to the 3-cycle $c = (3,4,5)$. The commutator of $a$ and $c$ is a 3-cycle. The commutator of generators is invariant under automorphisms of $F_2$, so the maximal characteristic subgroups $H_1$ and $H_2$ containing the kernels of $f_1$ and $f_2$ are different.

Now consider any word $w(X_1, X_2, \dots, X_N)$ whose image is contained in $H_1$.Equivalently, in $F_2/H_1$, this word evaluates identically to $1$. This happens ifand only if the projection to each $A_5$ factor evaluates identically to 1.Therefore, it also evaluates identically to $1$ in $F_2/H_2$. Any verbal subgroupcontained in $H_1$ is in fact contained in the intersection $H_1 \cap H_2$.

To get the actual maximal verbal subgroup contained in the kernel of an epimorphismsuch as $f_1$,first, form choose a representative for each orbit of the action of $Aut(A_5)$ on the product of copies of $Q$ A_5$for each of the$|Q|^n$homomorphisms$F_n F_2 \rightarrow Q$A_5$, and let $P$ be form the imageproduct of these homomorphisms, and let $P'$ P$be the subgroup of image.$P$generated by the orbit of the original quotient map under the automorphism group of$F_n$. We have comes equipped with a set pair of$n$generatorsfor$P$; ; let$R$be a set of relators, and let$S$be a set of words that generate$P/P'$. . Then (if I'm not mistaken)$R \cup S$R$ is a verbal description for the characteristic subgroup.

The main observation: this is inescably a very large set kernel of very long words. I think their interest is more theoretical than practicalthis map.These are rather opposite from verbal quotients defined by small sets of short words, such as $X^n$, or $[X,[X,Y]]$

3 Explained size estimate for characteristic subgroup, said some verbal words

Epimorphisms $F_n \rightarrow G$ are equivlaent to $n$-tuples of generators of elements of $G$ that generate. An $n$-tuple that does not generate $G$ must be contained in a proper subgroup; for a finite simple group, there are not many maximal subgroups of small index, so the vast majority of $n$-tuples actually generate.If there's only one orbit, the orbit size is rougly $|G|^n/|Aut(G)|$; for nonabelian finite simple groups, the outer automorphism group is small, so the orbit size is $|G|^{(n-1)}$ up to a small constant.

The corresponding characteristic subgroup is the kernel of thehomomorphism $F_n \rightarrow \prod G_i$, so this gives where the product is over all epimorphisms to $G$ in the orbit, up to automorphism of $G$. A subgroup of the product of two groups is the kernel of the map of the product to a common quotient; using this, since the factors are simple, this homomorphism is an epimorphism. Therefore,the characteristic subgroup of has huge index,

Even for $n = 2$, each orbit tends to be the orbits are fairly large, so the corresponding characteristic subgroup has have huge index.

I.e. characteristic subgroups with nonabelian simple composition factors are very sparse. Thus characteristic Characteristic subgroups with polycyclic factor group are the vast majority--- both because , for several reasons: there are a lot of polycyclic groups, and because they tend to have large automorphism groups,and different polycyclic quotients of a free group tend to correlate, so they determine the homomorphism to a characteristic subgroup product of a number of polycyclic quotients is often far from surjective. Thus,these characteristic subgroups do not quite as have such a huge an index. I suspect that in fact characteristic subgroups with abelian quotient should dominate. If so, the answer is boring, since

Verbal subgroups

A word $w$ in letters $X_1, \dots, X_k$ defines a function $f_w: G^k \rightarrow G$, when $k$-tuples of elements of $G$ are substituted for the $X_i$. As explained in comments, a verbal subgroup $H_V \subset G$ is one that is generated by the images of $f_w$, for $w$ in some set $V$.In other words, $G/H_V$ is what you get by the identities for $w \in V$ that $\forall (X_1, \dots X_k)\; \; w = 1$ (as well as the relations of $G$ if $G$ is not free).

Here's how to give a verbal descriptionof the characterisitic subgroup associated with a simple quotient $Q$ of $F_n$:first, form the product of copies of $Q$ for each of the $|Q|^n$homomorphisms $F_n \rightarrow Q$, and let $P$ be the image, and let $P'$ bethe subgroup of $P$ generated by the orbit of the original quotient map under the automorphism group of $F_n$. We have aset of $n$ generators for $P$; let $R$ be a set of relators, and let $S$ be a set of words that generate $P/P'$. Then (if I'm not mistaken) $R \cup S$ is a verbal description for the characteristic subgroup.

The main observation: this is inescably a very large set of very long words. I think their interest is more theoretical than practical. These are rather opposite from verbal quotients defined by small sets of short words, such as $X^n$, or $[X,[X,Y]]$

2 Corrected exponent formatting
1