show/hide this revision's text 5 added 1553 characters in body

As other people in this thread have pointed out, it's unsatisfying to make an automorphism tower that only stabilizes transfinitely as a direct limit, when all of the finite terms of the tower are abstractly isomorphic to the base group $G$. I Googled around a bit more and came back to the same two sources, Thomas' book, and this time a joint result of Hamkins and Thomas which is in chapter 8 of the book.

If an automorphism tower stabilizes after exactly $n \in \mathbb{N}$ steps in the direct limit sense, then it also stabilizes after exactly $n$ steps in the weaker abstract isomorphism sense. (Otherwise the direct limit "wouldn't know to stop".) Hamkins and Thomas do better than that. For any two ordinals $\alpha$ and $\beta$, which may or may not be finite numbers, they find one group $G$ whose automorphism tower has height $\alpha$ and $\beta$ in two different models of ZFC set theory. (Whether it's really the "same" group in different worlds is unclear to me, but their models are built to argue that it is so.) I would suppose that it is possible to make a tower without isomorphic terms by taking a product of these groups, even without the two-for-one property.

Other than one paper on the Grigorchuk group by Bartholdi and Sidki, I haven't found anything on automorphism towers of finitely generated groups. The Grigorchuk group has a countably infinite tower, but I'd have to learn more to know whether the terms are abstractly isomorphic.
show/hide this revision's text 4 More clarification; added 1 characters in body

I remember that my old grad classmate from Berkeley, Joel Hamkins, worked on the transfinite version of this problem. The Automorphism Tower Problem, by Simon Thomas, is an entire book on this subject. The beginning of the book gives the example of the infinite dihedral group $D_\infty$, in the sense of $\mathbb{Z}/2 \ltimes \mathbb{Z}$. It says that the automorphism tower of this group has height $\omega+1$. It also treats Joel's theorem, which says that every automorphism tower does stabilize, transfinitely. A Proceedings paper with the same author and title says that Wielandt showed that every finite centerless group has a finite automorphism tower.

An improved answer: Simon's book later shows that the automorphism tower of the finite group $D_8$ has height $\omega+1$, and that for general finite groups no one even knows a good transfinite bound. (The $8$ may look like a typo for $\infty$, but it's not :-).) Apparently the centerless condition is essential in Wielandt's condition.

Also, to clarify what these references mean by the automorphism tower, they specifically use the direct limit of the conjugation homomorphisms $G \to \mbox{Aut}(G)$. $D_8$ is abstractly isomorphic to its automorphism group. This is a different version of the question that I suppose does not have a transfinite extension. Section 5 of Thomas' book implies that it's an open problem whether the tower terminates in this weaker sense, for finite groups.

Finally an arXiv link to Joel Hamkins' charming paper, Every group has a terminating transfinite automorphism tower.
show/hide this revision's text 3 arXiv is a better citation

I remember that my old grad classmate from Berkeley, Joel Hamkins, worked on the transfinite version of this problem. The Automorphism Tower Problem, by Simon Thomas, is an entire book on this subject. The beginning of the book gives the example of the infinite dihedral group $D_\infty$, in the sense of $\mathbb{Z}/2 \ltimes \mathbb{Z}$. It says that the automorphism tower of this group has height $\omega+1$. It also treats Joel's theorem, which says that every automorphism tower does stabilize, transfinitely. A Proceedings paper with the same author and title says that Wielandt showed that every finite centerless group has a finite automorphism tower.

An improved answer: Simon's book later shows that the automorphism tower of $D_8$ has height $\omega+1$, and that for general finite groups no one even knows a good transfinite bound. Apparently the centerless condition is essential in Wielandt's condition.

Also, a direct an arXiv link to Joel Hamkins' charming paper, Every group has a terminating transfinite automorphism tower.
show/hide this revision's text 2 added 274 characters in body
show/hide this revision's text 1