Suppose that $G$ and $H$ are groups (not isomorphic) and $G\ast H$ the free product. Let $Aut(G)$, $Aut(H)$ be the automorphism groups of $G$ and $H$. What is $Aut(G\ast H)$ ?

4$\begingroup$ Surely this depends on the particular $G$ and $H$... $\endgroup$ – Alex Kruckman Nov 18 '15 at 2:37

$\begingroup$ It is certainly not a straightforward function of $\text{Aut}(G)$ and $\text{Aut}(H)$. For example, take $G = \mathbb{Z}_2, H = \mathbb{Z}_3$. Then $G \ast H \cong PSL_2(\mathbb{Z})$, whose automorphism group is, I imagine, $PGL_2(\mathbb{Z})$. $\endgroup$ – Qiaochu Yuan Nov 18 '15 at 4:25

2$\begingroup$ Consider the case $G=H=\mathbb{Z}$. Then we have $Aut(G)=Aut(H)=\mathbb{Z}_2$, but $G\ast H=F_2$, which has a very complicated automorphism group. $\endgroup$ – Sh.M1972 Nov 18 '15 at 6:51

$\begingroup$ It's called "free product". It is also the amalgam (or amalgamated product, but not amalgamate) of the two groups over the trivial subgroup. $\endgroup$ – YCor Nov 18 '15 at 8:10

3$\begingroup$ @QiaochuYuan, your example isn't a great one, since $\mathrm{Out}(\mathbb{Z}/2)=1$ and $\mathrm{Out}(\mathbb{Z}/3)=\mathrm{Out}(PSL_2(\mathbb{Z}))=\mathbb{Z}/2$, so in the outer automorphism group does in fact arise in a simple way from the outer automorphism groups of the factors. $\endgroup$ – HJRW Nov 18 '15 at 10:12
This is not an answer, but it's too long for a comment and makes an important point that I hope is useful.
It's not enough to think of your group just as some free product. To understand its automorphism group you need to think of it as a free product in a canonical way, and for this you need the Grushko decomposition. Recall:
Grushko's decomposition theorem: If $G$ is finitely generated then
$G=G_1*\ldots* G_n*F_k$
where each $G_i$ is freely indecomposable (ie isn't $\mathbb{Z}$ and doesn't split as a free product) and $F_k$ is free of rank $k$. Furthermore, the $G_i$ are determined up to conjugacy (and reordering), and $k$ is determined.
The key thing to notice here is that the $F_k$ factor is not determined up to conjugacy, and therefore if $k>0$ then there are complicated automorphisms that arise essentially from the different possible ways to choose $F_k$. However, the picture is relatively simple if $k=0$.
A modern reference on the (outer) automorphism groups of free products is this nice paper of Guirardel and Levitt.

$\begingroup$ Excellent, thank you! Actually your answer is more then what i was hoping for ! $\endgroup$ – Ofra Nov 18 '15 at 10:24

$\begingroup$ In fact, even with only two non free factors, $A,B$ you have complicated automorphism called "partial conjugations" : let $b\in B$, then there exists an automorphism which fixes $B$ and conjugate $A$ to $bAb^{1}$ $\endgroup$ – Thomas Nov 18 '15 at 12:06

$\begingroup$ @Thomas, well, it depends what you mean by 'complicated'. $\endgroup$ – HJRW Nov 18 '15 at 13:01

$\begingroup$ It is an automorphism which is not inner, and does not come from $Aut(A)$, $Aut(B)$, an analog of Dehn twist. In fact the automorphism group of $A*B$ is generated by such automorphisms, $Aut(A), Aut(B)$, and inner automorphisms....In the case of $A*Z$, you must add the automorphism which is $Id$ on $A$ and map $t$ to $ta$ if $t$ is the generator of $Z$. The general case is obtained by mixing these sorts of automorphisms... $\endgroup$ – Thomas Nov 18 '15 at 15:01

$\begingroup$ @Thomas, on the contrary, it does 'come from ... $\mathrm{Aut}(B)$'. More precisely, it's the composition of an inner automorphism of $B$ (which, as you rightly observe, gives you a nontrivial outer automorphism of the free product) with an inner automorphism of $A*B$. I would classify such things as not 'complicated', though I concede that what one may find complicated is a matter of taste. (But the definition I have in mind is that an automorphism is 'not complicated' if it fixes a point in the deformation space of Grushko decompositions.) ... $\endgroup$ – HJRW Nov 18 '15 at 15:09
The known reference is FuchsRabinowitz (or FouxeRabinowitz, Mat. Sbornik, 8 (1940), pp 265276 and 9 (1941) 183220. The papers are in Russian and German , see MR0003413 for a review. A more recent reference is McCullough Miller, Symmetric automorphisms of free products. Mem. Amer. Math. Soc. 122 (1996), no. 582, viii+97 pp.(MR1329943 )

$\begingroup$ Do you know if there exists a proof of FouxeRabinowitz's description of the generators elsewhere in the literature? $\endgroup$ – AGenevois Nov 18 '18 at 9:31

$\begingroup$ Collins Zieschang, Rescuing the Whitehead method for free products.Math. Z. 185 (1984), no. 4, 487–504 gives a more precise result...than FR. $\endgroup$ – Thomas Nov 19 '18 at 12:12

$\begingroup$ Thank you, I will take a look at the article. $\endgroup$ – AGenevois Nov 19 '18 at 17:39