Let $C_n$ be the cyclic group of order $n$. Its automorphism group $Aut(C_n)$ is a group of order $\varphi(n)$ isomorphic to $(\mathbb{Z}/n\mathbb{Z})^{\times}$ the multiplicative group of integer modulo $n$. This last group is abelian but not always cyclic, the first non-cyclic being $Aut(C_8) \simeq C_2 \times C_2$. Moreover the iterated automorphism groups $Aut^m(C_n)$ are not always abelian, as $Aut^2(C_8) \simeq S_3$.
By watching the table here for the group structure of $(\mathbb{Z}/n\mathbb{Z})^{\times}$, we cannot expect an easy classification for the set of groups $Aut(C_n)$, but (Q1) what about the following set? $$\{Aut^m(C_n) \ | \ n \ge 1, m \ge 0 \}$$
If it is also non-easy, (Q2) what about the set of groups $Aut^m(C_n)$ which are isomorphic to $Aut^{m+1}(C_n)$? For $n \le 15$, it is exactly $\{C_1,S_3,D_8 \}$ as shown by the following table:
$$\scriptsize{ \begin{array}{c|c} G &C_1&C_2&C_3&C_4&C_5&C_6&C_7&C_8&C_9&C_{10}&C_{11}&C_{12}&C_{13}&C_{14}&C_{15} \newline \hline Aut(G) &-&C_1&C_2&C_2&C_4&C_2&C_6&C_2^2&C_6&C_4&C_{10}&C_2^2&C_{12}&C_6&C_2 \times C_4 \newline \hline Aut^2(G) &-&-&C_1&C_1&C_2&C_1&C_2&S_3&C_2&C_2&C_4&S_3&C_2^2&C_2&D_8 \newline \hline Aut^3(G) &-&-&-&-&C_1&-&C_1&-&C_1&C_1&C_2&-&S_3&C_1&- \newline \hline Aut^4(G) &- &- &-&-&-&-&-&-&-&-&C_1&-&-&-&- \end{array} }$$
Let $\alpha(n)$ be the smallest $m \ge 0$ such that $Aut^m(C_n) \simeq Aut^{m+1}(C_n)$. Then:
$$\scriptsize{ \begin{array}{c|c} n &1&2&3&4&5&6&7&8&9&10&11&12&13&14&15 \newline \hline \alpha(n) &0&1&2&2&3&2&3&2&3&3&4&2&2&3&2 \end{array} }$$
Surprisingly, for $n<15$ (but not for $n=15$), $\alpha(n)$ is exactly the number of iterations of the Carmichael lambda function $\lambda$ needed to reach $1$ from $n$ (OEIS A185816). (Q3) Why?
A specific OEIS sequence has just been created (A331921) providing the value of $\alpha(n)$ for $n<32$; in this case, $\alpha(n) \le 5$ and $\{\mathrm{Aut}^5(C_n) \ | \ n < 32 \} = \{C_1,S_3,D_8,D_{12},\mathrm{PGL}(2,7) \}$. A usual laptop cannot compute $\alpha(n)$ for $n \ge 32$ (you are welcome to contribute to this sequence), we just know that $\alpha(32) \ge 6$$\alpha(32) \ge 8$. Here is the sequence $|Aut^n(C_{32})|$ for $n \le 6$$n \le 8$:
$$\scriptsize{ \begin{array}{c|c}
n &0&1&2&3&4&5&6&7&8 \newline
\hline
|Aut^n(C_{32})| & 2^5&2^4&2^4&2^6&2^{7}3&2^{9}3&2^{11}3&2^{16}3&2^{41}3^3 5^1 7 \newline
\hline
\text{IdGroup}(Aut^n(C_{32})) & [32,1]&[16,5]&[16,11]&[64,138]&[384,17948]&[1536,?]&[6144,?]&?&?
\end{array} }$$
We can also consider the sequence of $n$ such that $\exists m \ge 0$ with $Aut^m(C_n) \simeq C_1$:
$1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 14, 18, 19, 22, 23, 27, 38, 46, 47, 54, 81, \dots$ (A117729).
It seems to be a (strict) subsequence of A179401, so (Q4) why $Aut^m(C_n) \simeq C_1$ $\Rightarrow$ $\varphi^2(n) = \lambda^2(n)$?
It is an open problem whether the sequence $Aut^n(G)$ stabilizes (see here). For a given finite group $G$ there are three possibilities:
- (1): $\exists m \ge 0 $ such that $Aut^{m+1}(G) \simeq Aut^{m}(G)$,
- (2): not (1) and $\exists m \ge 0$ such that $\exists r>0$ with $Aut^{m+r}(G) \simeq Aut^{m}(G)$,
- (3): not (1) and not (2), i.e., $\forall m \ge 0$ and $\forall r>0$ then $Aut^{m+r}(G) \not \simeq Aut^{m}(G)$.
The case (1) means that the sequence $(Aut^m(G))_m$ is constant for $m$ large enough, (2) means that it is periodic non-constant for $m$ large enough, and (3) means never periodic. The existence of finite groups in cases (2) or (3) is open. (Q5) Can we rule out the cases (2) and (3) for the cyclic groups?
If not, let redefine $\alpha(n)$ as the smallest $m \ge 0$ such that $\exists r>0$ with $Aut^{m}(C_n) \simeq Aut^{m+r}(C_n)$ if $C_n$ is in cases (1) or (2), and $\infty$ in the case (3).