First, Baumslag's result is for finitely generated groups only. HW already says that essentially the Out of a residually finite group can be ``arbitrary". Now if you take a Tarski monster with trivial Out, then the direct product of it with the residually finite group above gives a non-residualy finite group with an arbitrary Out. Another way is to use Minasyan's result from Minasyan, Ashot, Groups with finitely many conjugacy classes and their automorphisms. Comment. Math. Helv. 84 (2009), no. 2, 259–296. He constructed a 2-generated group with two conjugacy classes (hence simple and not residually finite) with arbitrary countable Out. So there is no connection between properties of Out and residual finiteness of the group.
With Aut the situation is more complicated since if the group is not res. finite and does not have center then Aut is not residually finite. On the other hand, there are non-residually finite f.g.groups with residualy finite Aut. For example, Anna Erschler, I think, constructed such a group as a central extension of a Grigorchuk group. See Erschler, Anna Not residually finite groups of intermediate growth, commensurability and non-geometricity. J. Algebra 272 (2004), no. 1, 154–172.