The intersection of all normalizers of subgroups of a group $G$ is called the norm of $G$. By a resut of Schenkman [E. Schenkman, On the norm of a group, Illinois J. Math., 7 (1960) 150-152]  the norm of a group  always lies in $Z_2(G)$ of the upper central series.
So the question has positive answer if for example the center $Z(G)$ of $G$ is trvial.

The Schenkman result has been improved by Cooper [C.D.H. Cooper, Power automorphisms of a group, Math. Z., 107 (1968) 335-356.] as follows:
an automorphism which leaves every subgroup of a group $G$ invariant induces the trivial automorphism in the central factor group $G/Z(G)$.