It is a theorem of Gopal Prasad (which I hope I am not misquoting...) that lattices in higher rank linear semisimple Lie groups have finite outer automorphism groups. Is there some other reasonable class of groups with this finiteness property?
Here is a result of Fr\'ed\'eric Paulin: a hyperbolic group with Kazhdan's property (T), has a finite outer automorphism group. See Outer automorphisms of hyperbolic groups and small actions on Rtrees. Arboreal group theory (Berkeley, CA, 1988), 331–343, Math. Sci. Res. Inst. Publ., 19, Springer, New York, 1991. 


The theorem Igor Belegradek mentions is a more general form of the theorem (due collectively to Bestvina, Feighn, Paulin, and Rips) that a oneended hyperbolic group has infinite outer automorphism group only if it splits over a twoended subgroup. See M. Bestvina, M. Feighn, Stable actions of groups on real trees. Invent. Math. 121 (1995), no. 2, 287321. 

