First, Baumslag's result is for finitely generated groups only. HW already says that essentially the Out of a residually finite group can be ``arbitrary". Now if you take a Tarski monster with trivial Out, then the direct product of it with the residually finite group above gives a non-residualy finite group with an arbitrary Out. So there is no connection between properties of Out and residual finiteness of the group.
With Aut the situation is more complicated since if the group is not res. finite and does not have center then Aut is not residually finite. On the other hand, there are non-residually finite f.g.groups with residualy finite Aut. For example, Anna Erschler, I think, constructed such a group as a central extension of a Grigorchuk group.