It is probably best to concentrate on groups $G$ with $Z(G) =1$. For example, a finite group $G$ which is a direct product of non-Abelian simple groups is determined by its automorphism group. For $G \cong {\rm Inn}(G)$ is isomorphic to a normal subgroup of $A = {\rm Aut}(G)$. I claim that $G \cong E(A)$ (which is the central product of all components-that is, quasisimple subnormal subgroups- of $A$). Since $G \cong {\rm Inn}(G)$ in a natural manner we may regard $G$ as a (normal) subgroup of $A$, and then we certainly have $G \leq E(A)$. Now suppose that $A$ has a component $L$ which is not contained in $G$. Then $[L,G] = 1$ by standard properties of components (just the three subgroups Lemma is needed). But $L$ is a subgroup of ${\rm Aut}(G)$, a contradiction. Hence $G$ is isomorphic to $E(A)$, which is a uniquely determined characteristic subgroup of $A$.
Later edit: To be strict, I realise that this does not answer the question as asked. If $G$ is as in my answer, and two or more of the simple direct factors of $G$ are isomorphic, or if the simple direct factors of $G$ are pairwise non-isomorphic but at least one direct factor has non-trivial outer automorphisms, the above argument shows that ${\rm Aut}(A) \cong A = {\rm Aut}(G)$, while $G \not \cong A$, as $G$ has non-trivial outer automorphisms in those cases.
Hence to answer the question as asked, we need to take $G$ to be a direct product of pairwise non-isomorphic non-Abelian simple groups, each of which has trivial outer automorphism group.