Short answer: a typical example is G=SL(2,5), H = Z(G) = Z/2Z. If G/H and H are coprime and satisfy the condition, the G = G/H × H is quite dull.

I'll assume you find this interesting, and want to read about it:

Let G be a group (finite is good), H be an abelian normal subgroup of G, and Q be the quotient group.

If σ is an automorphism of a group G such that σ(H) = H, then σ induces automorphisms on H and G/H=Q.

If σ(h)=h for all h in H, then σ(H) = H, so we are interested in those σ that are "invisible" as automorphisms both of H and of Q.

The paper's condition is that no automorphism is invisible, which should seem a reasonable crutch if you want to talk about automorphisms of G in terms of those of H and Q (here it helps if H is also characteristic).

What do invisible automorphisms look like?

Well every element of G can be written as a product qh for some q in Q and h in H. (q1h1)(q2h2) = (q1q2)(h1^q2 * h2 * ζ(q1,q2)) where ζ:Q×Q→H is a (set-theoretic) function called a 2-cocycle. If you are only interested in semidirect products, then ζ(q1,q2) = 1_H can be ignored.

What does σ do to qh? Well it takes products to products, and h to h, but it only takes q to another element of the same coset, qδ(q) where δ:Q→H is another set-theoretic function. Hence σ(qh) = qh*δ(q).

A good example to keep in mind here are the extra-special groups, like the dihedral group of order 8. They have lots of invisible automorphisms. For instance (x,y)→(x,yz) where x,y are the main generators and z=[x,y] generates the center.

Now clearly not all δ can work, since surely δ(q1q2) is related somehow to δ(q1) and δ(q2). Indeed σ(q1q2) = (q1q2)δ(q1q2), but it is also equal to σ(q1)σ(q2) = (q1δ(q1))(q2δ(q2)) = (q1q2)*( δ(q1)^q2 * δ(q1) * ζ(q1,q2) ).

Ignoring ζ for a moment, one gets the equation δ(q1*q2) = δ(q1)^q2 * δ(q1), which expresses the fact that δ:Q→H is a derivation of the Q-module H. It is not too hard to see that the implications are reversible, and derivations help to define automorphisms fixing H and Q.

Not ignoring ζ does not change things very much, as instead of the subgroup of derivations inside the abelian group of all functions from Q to H, you just take a coset of this subgroup determined by ζ.

At any rate, so in your semi-direct products the condition is that there are no non-identity derivations from Q to H; the only derivation should have δ(q)=1_H for all q in Q.

Now of course derivations can exist even when Q and H are coprime: Take G to be the non-abelian group of order 6, H to be its subgroup of order 3. Then δ:Q→H takes the non-identity element of Q to any one of the three elements of H, giving three derivations δ. Checking the automorphism group of G, it is easy to see that every automorphism must take H to H, and must act as the identity on Q, since Aut(Q) = 1. Of the six automorphisms, three act as inversion on H, so are not inivisible, but three must be invisible, one for each δ.

In particular, being cyclic of prime and coprime order is not sufficient. Your coprime condition does, however, severely limit the variety of invisible homomorphisms that are available: they must all be conjugations by elements of H.

An automorphism of G induced by conjugation by an element of H must act trivially on H, since H is abelian. It must act trivially on Q, since hH = 1_Q in Q. Hence every automorphism induced by an element of H is invisible.

If G is to have no invisible automorphisms, then H must be central, since q^h = h^-1 * q * h = q* (h^q)^-1 * h is only the identity on G if h^q = h for all q.

In other words, the paper's condition implies H is central, so you are looking for a group Q with a trivial module H whose first cohomology vanishes, but whose second does not. I think this is reasonably rare. Probably a very standard example is a perfect group Q that is not superperfect, and H to be its Schur multiplier.

For instance, take G=SL(2,5), Q = Alt(5), H = Z/2Z.