It seems that any distance-transitive graph satisfies your condition (of course, distance-transitivity is much stronger), and you can find many examples (both finite and infinite) in [Peter Cameron's nice paper.][1] (A census of infinite distance-transitive graphs, Discrete Math, 1998) Many of these are not Cayley graphs, it would appear.


  [1]: http://ac.els-cdn.com/S0012365X98000636/1-s2.0-S0012365X98000636-main.pdf?_tid=c4d99bb4-373e-11e5-85e0-00000aab0f02&acdnat=1438318023_0cc21cceda5922e9e5fdcb66430643b9