Periodic Automorphism Towers In Scott's classic textbook on Group Theory, he asks:
Suppose that $G$ is a finite group. Is the sequence of isomorphism types
of the groups $Aut^{(n)}(G)$ for $n \in \mathbb{N}$ eventually periodic?
Here $Aut^{(2)}(G) = Aut(Aut(G))$ etc. Equivalently, is the sequence
$|Aut^{(n)}(G)|$ always bounded above?
It apparently remains opens whether the sequence of automorphism types of $Aut^{(n)}(G)$ is in fact always eventually constant. (A wonderful theorem of Wielandt says that if $G$ is a finite centerless group, then the sequence is eventually constant.) So I would like to ask:
Does there exists a finite group such that $Aut(G) \not \cong G$ but
$Aut^{(n)}(G) \cong G$ for some $n \geq 2$?
Edit: Joel has pointed out that my question is perhaps even open for infinite groups. This sounds like an interesting question which doesn't seem amenable to the standard tricks.
 A: Remark: The comments below say that the group $G$ given
here is not an example, since ${\rm Aut}({\rm Aut}(G))$ is isomorphic to
${\rm Aut}(G)$ rather than to $G$.
I think this anwers the question for infinite groups:
MR0470091 (57 #9858)
Collins, Donald J.
The automorphism towers of some one-relator groups.
Proc. London Math. Soc. (3) 36 (1978), no. 3, 480--493.
20F55 
Theorem (ii) states that if $G=\langle a,b \mid a^{-1}b^ra=b^s \rangle$ is a Baumslag-Solitar group with $r-s$ even, then $Aut(Aut(G))$ is isomorphic to $G$ and $G$ has an outer automorphism. 
Moreover, when $r=1$, $G$ is the semidirect product $\mathbf Z \ltimes \mathbf Z[\frac 1 s]$, where $\mathbf Z$ acts via multiplication by $\frac 1 s$. Then $G$ is torsionfree, but $Aut(G)$ has an element of order 2 (see his lemma 3). If $G$ is represented as a matrix group, $(a,b) \mapsto \begin{pmatrix} s^a & b \\ 
0 & 1 \end{pmatrix}$, then this outer automorphism is explicitely given by conjugation by $diag(i,-i)$, where $i$ is a square root of -1.
A: I have been researching this question on my own time along with generally studying finite groups of small order in an attempt to better understand their structure, and I have come up with some results in those efforts that I haven't seen mentioned in this thread or the other similar ones I've found on this site.
I had attempted to see if I could prove a bound for $|Aut^{(n)}(G)|$ and had thought to use the elementary fact that for $|G|=n$, $Aut(G)$ is a subgroup of $S_n$. The only problem is that the way that fact is proved doesn't generalize to the realization of $Aut(G)$ in $S_n$, if one could show that for any subgroup $H \subset S_n$ that $Aut(H)$ is a subgroup of $Aut(S_n)$ then we would have our bound immediately (namely $|Aut^{(k)}(G)| \leq n!$).
Edit: It turns out this bound is not correct, as for $G=$SmallGroup(16,5), we have $|Aut^{(7)}(G)|>16!$, it is of order ~2 quadrillion, whereas 16! is about 20 trillion.
Something that might help with calculating automorphism groups in general and understanding how $Aut^{(k)}(G)$ determines $Aut^{(k+1)}(G)$ is a result I arrived at fairly recently that depends on an idea I've been interested in for a long time. The idea is that some elements of a given order in a group can be distinguished from one another by what role they play in the group structure, for instance the unique order 2 element in $D_8$ that commutes with all the other elements in $D_8$ is very distinct from the other 4 order 2 elements. The criterion I arrived at to capture this idea is that two elements of identical order belong to the same 'structural congruence class' if there is a lattice isomorphism between the sublattices of all the subgroups which contain the cyclic subgroup generated by the given element which respects the isomorphism classes of the subgroups in the given sublattices. As an example, returning to the order 2 elements in $D_8$, the remaining 4 elements belong to the same structural congruence class, they are all members of exactly 1 Klein four subgroup which is a subgroup of all of $D_8$, however there is an additional subtlety here, the 4 order 2 elements can be paired off by which Klein four subgroup they belong to (there are two Klein four subgroups of $D_8$, one is generated by the vertical and horizontal flips, the other by the diagonal flips). Combining this idea with an observation about the cyclic subgroups and generating sets yielded a very interesting result. The observation is that one can readily determine the minimum size of a generating set (and which sets will work) by analyzing the full subgroup lattice. The point is that the subgroup generated by two given elements is exactly the join of the cyclic subgroups they generate in the full subgroup lattice, hence in a 2-generated group, you will be able to find a pair of cyclic groups whose join is the entire group. In a 3-generated group, that does not occur, but a set of 3 cyclic subgroups will suffice, etc. You can also readily enumerate the full list of minimum size generating sets this way (and indeed you can usually use combinatorics to quickly count the possibilities once you find a few minimum size generating sets). Armed with a complete list of minimum size generating sets, you can use the fact that the action of any automorphism is determined by its action on a generating set, and that generator map to generators to determine the size of the automorphism group. You can only map a given generating set to one in which the elements are of the same classes as the ones in the generating set you are starting with. Returning to $D_8$, we see that the generating sets are those which contain an order 4 element and one of the 4 order 2 elements not generated by the order 4 elements as well as pairs of order 2 elements which do not commute (there are only 4 such pairs, since the order 2 elements not generated by the order 4 elements pair off based on membership in klein four subgroups and hence commute with each other, but not the whole group). These are two different 'classes' of generating sets. One of size 4, the other of size 8. In the smaller class, not only can you send the given generating set to any of the others, but you can also swap which generator is which, which gives 8 possible automorphisms. You can get these same automorphisms by determining where to send a generating set consisting of an order 4 and order 2 element instead. This procedure works in general (it is not terribly difficult to prove, though pinning down the precise definition of 'structural congruence class' may be tricky (notice in $D_8$ how the class with 4 order 2 elements further broke down into two 'subclasses' based on the pairs by whether or not they commuted with each other).
As final remarks, I have used GAP to compute $Aut^{(n)}(G)$ for all groups $G$ of order less than 32 up to the largest $n$ that GAP would compute in a reasonable amount of time (usually only 4-5 steps or until stabilization). Thos groups for which I have yet to show stabilization definitely undergo a serious combinatorial explosion. For instance the $G=SmallGroup(16,3)$ has
$Aut(G) \simeq SmallGroup(32,27)$
$Aut^{(2)}(G) \simeq SmallGroup(384,20100)$
$Aut^{(3)}(G) \simeq $ a group of order 2304 with 8 generators
$Aut^{(4)}(G) \simeq $ a group of order 110592 with 7 generators
This last group took roughly 15 seconds for GAP to even construct, asking it to give a structure description took over 10 minutes and required nearly 2GB of RAM (!!).
Asking it to find $Aut^{(5)}(G)$ was unsuccessful even after 5 hours of computation and again using nearly 2GB of RAM (creating $Aut^{(4)}(G)$ only took a about 150MB, though the structure description took WAY more).
I just completed computing $Aut^{(4)}(G)$ for SmallGroup(16,5), which took about an hour, though the group is of only of order 1536 (no doubt the structural complexity of groups with a high prime power in their order had something to do w/ that, it also took nearly 600MB of RAM to do it), and I haven't even asked for a structure description.
