It seems that any distance-transitive graph satisfies your condition (of course, distance-transitivity is much stronger), and you can find many examples (both finite and infinite) in [Peter Cameron's nice paper.][1] (A census of infinite distance-transitive graphs, Discrete Math, 1998) Many of these are not Cayley graphs, it would appear.


  [1]: https://core.ac.uk/download/pdf/82631488.pdf