Consider the Kneser graph $K(v,k)$, with vertices the $k$-subsets of $V=\{1,\ldots,v\}$, where the $k$-subsets are adjacent if they are disjoint. Let $\alpha=\{1,\ldots,k\}$ and let $G$ be the subgroup of the symmetric group on $V$ that fixes each element of the complement of $\alpha$. So $|G|=k!$ and $G$ is a subgroup of the stablizer of $\alpha$ that fixes each neighbour of $\alpha$ in the Kneser graph.

To get examples as you requested, take $k>3$ (and $v>2k)$.

Cayley graphs are not a good place to look, because the stabilizer of a vertex tends to consist of automorphisms of the underlying group, and any automorphism that fixes each element in a connection set is the identity.

Consider the Kneser graph $K(v,k)$, with vertices the $k$-subsets of $V=\{1,\ldots,v\}$, where the $k$-subsets are adjacent if they are disjoint. Let $\alpha=\{1,\ldots,k\}$ and let $G$ be the subgroup of the symmetric group on $V$ that fixes each element of the complement of $\alpha$. So $|G|=k!$ and $G$ is a subgroup of the stablizer of $\alpha$ that fixes each neighbour of $\alpha$ in the Kneser graph.

To get examples as you requested, take $k\ge 3$ (and $v>2k)$.

Cayley graphs are not a good place to look, because the stabilizer of a vertex tends to consist of automorphisms of the underlying group, and any automorphism that fixes each element in a connection set is the identity.