show/hide this revision's text 3 added 1 characters in body

I think this anwers the question for infinite groups:

MR0470091 (57 #9858) Collins, Donald J. The automorphism towers of some one-relator groups. Proc. London Math. Soc. (3) 36 (1978), no. 3, 480--493. 20F55

Theorem (ii) states that if $G=\langle a,b \mid a^{-1}b^ra=b^s \rangle$ is a Baumslag-Solitar group with $r-s$ even, then $Aut(Aut(G))$ is isomorphic to $G$ and $G$ has an outer automorphism.

Moreover, when $r=1$, $G$ is the semidirect product $\mathbf Z \ltimes \mathbf Z[\frac 1 s]$, where $\mathbf Z$ acts via multiplication by $\frac 1 s$. Then $G$ is torsionfree, but $Aut(G)$ has an element of order 2 (see his lemma 3). If $G$ is represented as a matrix group, $(a,b) \mapsto \begin{pmatrix} s^a & b \\ 0 & 1 \end{pmatrix}$, then this outer automorphis automorphism is explicitely given by conjugation by $diag(i,-i)$, where $i$ is a square root of -1.
show/hide this revision's text 2 added 312 characters in body; added 20 characters in body

I don't know if think this answers anwers the question , but certainlyfor infinite groups:

MR0470091 (57 #9858) Collins, Donald J. The automorphism towers of some one-relator groups. Proc. London Math. Soc. (3) 36 (1978), no. 3, 480--493. 20F55

is an important paper in this direction:

Theorem (ii) states that if $G=\langle a,b \mid a^{-1}b^ra=b^s \rangle$ is a Baumslag-Solitar group with $r-s$ even, then $Aut(Aut(G))$ is isomorphic to $G$ and $G$ has an outer automorphism.

(I don't know, though

Moreover, if Collins claims that when $G r=1$, $G$ is the semidirect product $\mathbf Z \not\cong ltimes \mathbf Z[\frac 1 s]$, where $\mathbf Z$ acts via multiplication by $\frac 1 s$. Then $G$ is torsionfree, but $Aut(G)$ in this case)has an element of order 2 (see his lemma 3). If $G$ is represented as a matrix group, $(a,b) \mapsto \begin{pmatrix} s^a & b \\ 0 & 1 \end{pmatrix}$, then this outer automorphis is explicitely given by conjugation by $diag(i,-i)$, where $i$ is a square root of -1.
show/hide this revision's text 1

I don't know if this answers the question, but certainly

MR0470091 (57 #9858) Collins, Donald J. The automorphism towers of some one-relator groups. Proc. London Math. Soc. (3) 36 (1978), no. 3, 480--493. 20F55

is an important paper in this direction: Theorem (ii) states that if $G=\langle a,b \mid a^{-1}b^ra=b^s \rangle$ is a Baumslag-Solitar group with $r-s$ even, then $Aut(Aut(G))$ is isomorphic to $G$ and $G$ has an outer automorphism.

(I don't know, though, if Collins claims that $G \not\cong Aut(G)$ in this case).