Does $\mathrm{Aut}(\mathrm{Aut}(...\mathrm{Aut}(G)...))$ stabilize? Purely for fun, I was playing around with iteratively applying $\DeclareMathOperator{\Aut}{Aut}\Aut$ to a group $G$; that is, studying groups of the form
$$ {\Aut}^n(G):= \Aut(\Aut(...\Aut(G)...)) $$
Some quick results:


*

*For finitely-generated abelian groups, it isn't hard to see that this sequence eventually arrives at trivial group.

*For $S_n$, $n\neq 2,6$, the group has no center and no outer automorphisms, and so the conjugation action provides an isomorphism $G\simeq \Aut(G)$.

*Furthermore, I believe that if $G$ is a non-abelian finite simple group, then $\Aut(\Aut(G))\simeq \Aut(G)$, though this is based on hearsay.

*If one considers topological automorphisms of topological groups, then $\Aut(\mathbb{R})\simeq \mathbb{Z}/2\mathbb{Z}\times \mathbb{R}$, and so $$\Aut(\Aut(\mathbb{R}))\simeq \Aut(\mathbb{Z}/2\mathbb{Z}\times \mathbb{R})\simeq\mathbb{Z}/2\mathbb{Z}\times \mathbb{R}.$$


When the sequence $\Aut^n(G)$ is constant for sufficiently large $n$, we will say the sequence stabilizes.  Despite my best efforts, I have been unable to find a group $G$ such that the sequence $\Aut^n(G)$ is provably non-stabilizing.  Is this possible?
A slightly deeper question is whether there are groups $G$ such that the sequence becomes periodic after some amount of time.  That is, $\Aut^n(G)\simeq \Aut^{n+p}(G)$ for some $p$ and for $n$ large enough, but $\Aut^n(G)\not\simeq \Aut^{n+m}(G)$ for $m$ between 0 and $p$.  A simple way to produce such an example would be to give two groups $G \neq H$ such that $\Aut(G)\simeq H$ and $\Aut(H)\simeq G$.  Does anyone know an example of such a pair?
 A: "if G is a non-abelian simple group, then Aut(Aut(G))=Aut(G)."
This is Theorem 7.14 (Page 162) of 
Rotman, Joseph J. (1995), An introduction to the theory of groups,
Berlin, New York: Springer-Verlag, ISBN 978-0-387-94285-8
© 1995 Springer-Verlag New York, Inc.
ISBN 0-387-94285-8 Springer-Verlag New York Berlin Heidelberg
ISBN 3-540-94285-8 Springer-Verlag Berlin Heidelberg New York
All the best,
Fernando.
A: $\DeclareMathOperator{\Aut}{Aut}$I don't know about non-stabilizing, but rigidity provides many examples that stabilize quickly.
1) Let $\pi$; be the fundamental group of a finite volume hyperbolic manifold $M$ of dimension $\ge 3$ with no symmetries (that is, no nontrivial self-isometries). Negative curvature implies that $\pi$; is centerless, so the map $\pi\to \Aut(\pi)$ is injective. Mostow-Prasad rigidity says that $\operatorname{Out}(\pi) = \operatorname{Isom}(M)$, so the lack of isometries implies that $\operatorname{Out}(\pi)$ is trivial and $\Aut(\pi) = \pi$. [This works verbatim for lattices in higher-rank semi-simple Lie groups subject to appropriate conditions.]
2) Let $\pi=F_d$ be a free group of rank $2\le d\lt \infty$. Then $\Aut(F_n)$ is a much larger group; however, Dyer-Formanek showed that $\operatorname{Out}(\Aut(F_n))$ is trivial. Thus since $\Aut(F_n)$ is clearly centerless, we have $\Aut(\Aut(F_n)) = \Aut(F_n)$.
3) Interpolating between these two examples, if $\pi=\pi_1(S_g)$ is the fundamental group of a surface of genus $g\ge 2$, then $\Aut(\pi)$ is the so-called "punctured mapping class group" $\text{Mod}_{g,^*}$, which is much bigger than $\pi$. Ivanov proved that $\operatorname{Out}(\text{Mod}_{g,^*})$ is trivial, and since $\text{Mod}_{g,^*}$ is again centerless, we have $\Aut(\Aut(\pi_1(S_g))) = \Aut(\pi_1(S_g))$.
In each of these cases, rigidity in fact gives stronger statements: Let $H$ and $H'$ be finite index subgroups of $G = \Aut(F_n)$ or $\text{Mod}_{g,^*}$. (This class of groups can be widened enormously, these are just some examples.) Then any isomorphism from $H$ to $H'$ comes from conjugation by an element of $G$, by Farb-Handel and Ivanov respectively. In particular, $\Aut(H)$ is the normalizer of $H$ in $G$. Rigidity gives the same conclusion for $H = \pi_1(M)$ as in the first example and $G = \operatorname{Isom}(H^n)$ [which is roughly $\operatorname{SO}(n,1)$]. It seems that by carefully controlling the normalizers, you could use this to construct examples that stabilize only after $n$ steps, for arbitrary large $n$.

Edit: I find the examples of $D_8$ and $D_\infty$ unsatisfying because even though $\operatorname{Inn}(D)$ is a proper subgroup of $\Aut(D)$, we still have $\Aut(D)$ isomorphic to $D$. Here is a general recipe for building similarly liminal examples. Let $G$ be an infinite group with no $2$-torsion so that $\Aut(G) = G$ and $H^1(G;\Bbb{Z}/2\Bbb{Z}) = \Bbb{Z}/2\Bbb{Z}$. (Edited: For example, by rigidity, any hyperbolic knot complement with no isometries has these properties; by Thurston, most knot complements are hyperbolic.) The condition on the $2$-torsion implies that for any automorphism $G \times \Bbb{Z}/2\Bbb{Z} \to G \times \Bbb{Z}/2\Bbb{Z}$, the composition
$$G \to G \times \Bbb{Z}/2\Bbb{Z} \to G \times \Bbb{Z}/2\Bbb{Z} \to G$$
is an isomorphism. From this we see that $\Aut(G \times \Bbb{Z}/2\Bbb{Z}) / G = H^1(G;\Bbb{Z}/2\Bbb{Z}) = \Bbb{Z}/2\Bbb{Z}$. By examination the extension is trivial, and thus $\Aut(G \times \Bbb{Z}/2\Bbb{Z}) = G \times \Bbb{Z}/2\Bbb{Z}$. However, the image $\operatorname{Inn}(G \times \Bbb{Z}/2\Bbb{Z})$ is the proper subgroup $G$.
Comments: looking back, this feels very close to your original example of $\Bbb R \times \Bbb Z/2\Bbb Z$. Interesting that it's (seemingly) much harder to find group-theoretic conditions to force the behavior the way you want, while topologically it's easy.
Also, if you instead take $G$ with $H^1(G;\Bbb Z/2\Bbb Z)$ having larger dimension, say $H^1(G;\Bbb Z/2\Bbb Z) = (\Bbb Z/2\Bbb Z)^2$, this blows up quickly. You get $\Aut(G \times \Bbb Z/2 \Bbb Z) = G \times (\Bbb Z/2\Bbb Z)^2$, but then $\Aut(\Aut(G \times  \Bbb Z/2\Bbb Z))$ is the semidirect product of $H^1(G;(\Bbb Z/2\Bbb Z)^2) = (\Bbb Z/2\Bbb Z)^4$ with $\Aut(G) \times \Aut((\Bbb Z/2\Bbb Z)^2) = G \times \operatorname{GL}(2,2)$. Already the next step seems very hard to figure out. However, if you had enough control over the finite quotients of $G$, perhaps you could show that the linear parts of these groups don't get "entangled" with the rest, so that the automorphism groups would act like a product of $G \times (\Bbb Z/2\Bbb Z)^n$ with something else, with n going to infinity. If so, this could yield an example where the isomorphism types of the groups never stabilize.
A: I remember that my old grad classmate from Berkeley, Joel Hamkins, worked on the transfinite version of this problem.  The Automorphism Tower Problem, by Simon Thomas, is an entire book on this subject.  The beginning of the book gives the example of the infinite dihedral group $D_\infty$, in the sense of $\mathbb{Z}/2 \ltimes \mathbb{Z}$.  It says that the automorphism tower of this group has height $\omega+1$.  It also treats Joel's theorem, which says that every automorphism tower does stabilize, transfinitely.  A Proceedings paper with the same author and title says that Wielandt showed that every finite centerless group has a finite automorphism tower.

An improved answer:  Simon's book later shows that the automorphism tower of the finite group $D_8$ has height $\omega+1$, and that for general finite groups no one even knows a good transfinite bound.  (The $8$ may look like a typo for $\infty$, but it's not :-).) Apparently the centerless condition is essential in Wielandt's condition.
Also, to clarify what these references mean by the automorphism tower, they specifically use the direct limit of the conjugation homomorphisms $G \to \mbox{Aut}(G)$.  $D_8$ is abstractly isomorphic to its automorphism group.  This is a different version of the question that I suppose does not have a transfinite extension.  Section 5 of Thomas' book implies that it's an open problem whether the tower terminates in this weaker sense, for finite groups.
Finally an arXiv link to Joel Hamkins' charming paper, Every group has a terminating transfinite automorphism tower.

As other people in this thread have pointed out, it's unsatisfying to make an automorphism tower that only stabilizes transfinitely as a direct limit, when all of the finite terms of the tower are abstractly isomorphic to the base group $G$.  I Googled around a bit more and came back to the same two sources, Thomas' book, and this time a joint result of Hamkins and Thomas which is in chapter 8 of the book.
If an automorphism tower stabilizes after exactly $n \in \mathbb{N}$ steps in the direct limit sense, then it also stabilizes after exactly $n$ steps in the weaker abstract isomorphism sense.  (Otherwise the direct limit "wouldn't know to stop".)  Hamkins and Thomas do better than that.  For any two ordinals $\alpha$ and $\beta$, which may or may not be finite numbers, they find one group $G$ whose automorphism tower has height $\alpha$ and $\beta$ in two different models of ZFC set theory.  (Whether it's really the "same" group in different worlds is unclear to me, but their models are built to argue that it is so.)  I would suppose that it is possible to make a tower without isomorphic terms by taking a product of these groups, even without the two-for-one property.
Other than one paper on the Grigorchuk group by Bartholdi and Sidki, I haven't found anything on automorphism towers of finitely generated groups.  The Grigorchuk group has a countably infinite tower, but I'd have to learn more to know whether the terms are abstractly isomorphic.
A: I had a group theory course with Suzuki in 1993, and he proved that 


*

*if G is a non-abelian finite simple group, then Aut(Aut(G))=Aut(G).


I don't have that proof any longer, so this is still just hearsay, but with pedigree!
A: For reference, note that the Wielandt Automorphism Tower Theorem states:

Let G be a finite group, and assume that Z(G) = 1. Write G1 = G, and for i > 1, Gi = Aut(Gi - 1). Then up to isomorphism, there are only finitely many different groups among the Gi.

In fact, I quote the above result from Martin Isaacs' Finite Group Theory; more precisely, the result can be found on page 278 in Chapter 9 of this publication. Isaacs discusses this result in the context of subnormality in some depth and proves it as well.
A: It seems that $Aut(G)$ is complete for any simple (not necessarily finite) group. The proof of this is in Rotman's book, but it is very short. 
A: I proved with Denis Osin in arXiv:math/0605553 that if G is a (word) hyperbolic group with property (T), then Aut(G) contains Inn(G)=G as a subgroup of finite index. So Aut(G) is hyperbolic and has (T), and we can iterate to get an automorphism tower in which each group is hyperbolic and has (T). A question I like (even though this seems idle curiosity) is whether this automorphism tower always stabilizes. I have no clue how to approach it.
