There are many, many examples of large, residually finite groups with trivial outer automorphism group. Indeed, any nonabelian virtually special group is large and residually finite. Furthermore, if they're hyperbolic, Sela Paulin showed that they have finite outer automorphism group unless they split over a virtually cyclic subgroup; in particular, most such examples have trivial outer automorphism group. Instances of this include:
fundamental groups of closed hyperbolic 3-manifolds;
random groups at density less than 1/6;
most small-cancellation groups;
groups constructed using the Rips construction;
etc etc.