Suppose H is a subgroup of a finite group G. Can the group of all automorphisms of H that extend to G can be characterized somehow? What condition on the group extension would guarantee that any automorphism of H can be extended to G?

An abstract answer to the question for all groups is given in the papers below. I have not followed the field in recent years. There may be other papers specific to finite groups. Charles Wells, Automorphisms of Group Extensions, 1970. Kung Wei Yang Isomorphisms of group extensions. Pacific J. Math. Volume 50, Number 1 (1974), 299304. D.J.S. Robinson, Applications of cohomology to the theory of groups, Groups – St. Andrews 1981, London Math. Soc. Lecture Notes vol. 71 (1982), pp. 46–80. Jin Ping, Automorphisms of groups Journal of Algebra Volume 312, Issue 2, 15 June 2007, Pages 562569 


The papers in the accepted answer by @SixWingedSeraph all refer to a somewhat more specific problem, that of extending an automorphism of a normal subgroup to a larger group. Although the questioner says something about a group extension (which would usually imply normality of $H$), he did not specify that $H$ should be normal. Extending automorphisms of normal subgroups I haven't managed to get my hands on the Robinson paper "Applications of cohomology to the theory of groups", but I understand it has some explanation and applications of Wells' result, as well as a new proof. The Jin paper "Automorphisms of groups" restates the result of Wells in somewhat different language, and applies it to the case where one wants to find an automorphism of $N$ which acts trivially on $G/N$. I'll remark that one interesting case is where $N$ is the direct product of isomorphic finite simple groups (e.g., a nonabelian chief factor). Here $Z(N)$ is trivial, which makes the long exact sequence above collapse rather nicely. But in this case one can obtain similar results by using the fact that if $N$ is center free, then any homomorphism $\alpha : Q \rightarrow \operatorname{Out}(N)$ uniquely determines an extension of $Q$ by $N$. This is however only mildly more elementary, as proving this fact also requires group cohomology techniques  see chapter IV.6 of Ken Brown's book "Cohomology of groups". Extending automorphisms of arbitrary subgroups So I guess the overall answer to the original question is "sometimes not". If you have a specific situation which is still of interest, you might try looking at the second Pettet paper to see if it helps you show the automorphism extends or does not extend. It looks like it would be a hard problem to find a general characterization. 

