Characterising extendable automorphisms

Question

Suppose H is a subgroup of a finite group G. Can the group of all automorphisms of H that extend to G can be characterized somehow? What condition on the group extension would guarantee that any automorphism of H can be extended to G?

Answer 1

An abstract answer to the question for all groups is given in the papers below. I have not followed the field in recent years. There may be other papers specific to finite groups.

Charles Wells, Automorphisms of Group Extensions, 1970.

Kung Wei Yang Isomorphisms of group extensions. Pacific J. Math. Volume 50, Number 1 (1974), 299-304.

D.J.S. Robinson, Applications of cohomology to the theory of groups, Groups – St. Andrews 1981, London Math. Soc. Lecture Notes vol. 71 (1982), pp. 46–80.

Jin Ping, Automorphisms of groups Journal of Algebra Volume 312, Issue 2, 15 June 2007, Pages 562-569

Answer 2

The papers in the accepted answer by @SixWingedSeraph all refer to a somewhat more specific problem, that of extending an automorphism of a normal subgroup to a larger group. Although the questioner says something about a group extension (which would usually imply normality of $H$), he did not specify that $H$ should be normal.

Extending automorphisms of normal subgroups
The Wells paper "Automorphisms of group extensions" is the first paper I know about on this topic. In the situation where $N$ is a normal subgroup of $G$, such that $\alpha : G/N \rightarrow \operatorname{Out}(N)$ is the action of the quotient on $N$, Wells finds an exact sequence $$1 \rightarrow Z^1_\alpha(G/N, Z(N)) \rightarrow \operatorname{Aut}(G; N) \rightarrow \operatorname{Compat}(G/N; N) \rightarrow H^2_\alpha(G/N, Z(N)). $$ Here $\operatorname{Aut}(G; N)$ is the group of all automorphisms of $G$ fixing $N$, and $\operatorname{Compat}(G/N; N)$ is all pairs of automorphisms of the quotient $G/N$ and subgroup $N$ satisfying a certain compatibility condition. $Z^1$ and $H^2$ are the 1st cocycle space and the 2nd group cohomology, respectively. It should be noticed that the last map is not a group homomorphism.

I haven't managed to get my hands on the Robinson paper "Applications of cohomology to the theory of groups", but I understand it has some explanation and applications of Wells' result, as well as a new proof. The Jin paper "Automorphisms of groups" restates the result of Wells in somewhat different language, and applies it to the case where one wants to find an automorphism of $N$ which acts trivially on $G/N$.

I'll remark that one interesting case is where $N$ is the direct product of isomorphic finite simple groups (e.g., a nonabelian chief factor). Here $Z(N)$ is trivial, which makes the long exact sequence above collapse rather nicely. But in this case one can obtain similar results by using the fact that if $N$ is center free, then any homomorphism $\alpha : Q \rightarrow \operatorname{Out}(N)$ uniquely determines an extension of $Q$ by $N$. This is however only mildly more elementary, as proving this fact also requires group cohomology techniques -- see chapter IV.6 of Ken Brown's book "Cohomology of groups".

Extending automorphisms of arbitrary subgroups
Skupp shows in "A characterization of inner automorphisms" that an automorphism $\varphi$ of a group $H$ extends to an automorphism in every group $G$ with $H$ embedded in $G$ if and only if $\varphi$ is inner (i.e., obtained via conjugation by some element). Pettet showed that the same holds if we restrict $G$ to be finite in "On inner automorphisms of finite groups" and "Characterizing inner automorphisms of groups".

So I guess the overall answer to the original question is "sometimes not". If you have a specific situation which is still of interest, you might try looking at the second Pettet paper to see if it helps you show the automorphism extends or does not extend. It looks like it would be a hard problem to find a general characterization.