This is a sort of a follow-up question to Limits of conjugated subgroups (though it might not seem at first glance to have much to do with it.)

Anyway, I'm wondering what sort of groups have the property that the centralizer of an element of infinite order is virtually abelian. For example hyperbolic groups are known to have this property: the infinite cyclic subgroup $\langle g\rangle$ has finite index in $C(g)$.

That's about the extent of my knowledge in this question.

In particular I'm interested in groups acting by isometries on a CAT(0) space (especially fundamental groups of NPC-spaces), semihyperbolic groups etc.

Of course, any information someone might be able to provide is greatly appreciated.