The Mathieu group $M_{11}$ does not have this property. A quote from Example 2.16 in this paper: "Hence there is no automorphism of $M_{11}$ that maps $x$ to $x^{−1}$."
Background how I found this quote as I am no group theorist: I used Google on "groups with no outer automorphism" which led me to this Wikipedia article, and from there I jumped to this other Wikipedia article. So I learned that $M_{11}$ has no outer automorphism. Then I used Google again on "elements conjugate to their inverse in the mathieu group" which led me to the above mentioned paper.
EDIT: Following Geoff Robinson's comment let me show that any element $x\in M_{11}$ of order 11 has this property, using only basic group theory and the above Wikipedia article. The article tells us that $M_{11}$ has 7920 elements of which 1440 have order 11. So $M_{11}$ has 1440/10=144 Sylow 11-subgroups, each cyclic of order 11. These subgroups are conjugates to each other by one of the Sylow theorems, so each of them has a normalizer subgroup of order 7920/144=55. In particular, if $x$ and $x^{-1}$ were conjugate to each other, then they were so by an element of odd order. This, however, is impossible as any element of odd order acts trivially on a 2-element set.
