I suspect that the answer to my question is well-known to be no. To be more precise, let $G$ and $H$ be nonisomorphic finite groups of the same order. Let $S \subseteq G$ and $T \subseteq H$ be subsets satisfying the three properties: (1) the subsets are symmetric, that is $S = S^{-1}$ and $T = T^{-1}$; (2) they are minimal symmetric generating sets; (3) the size of $S$ is equal to the size of $T$. Is it possible for the Cayley graph of the pair $(G,S)$ and the Cayley graph of the pair $(H,T)$ to be isomorphic?
If the answer is yes, what is the smallest such example?