This `answer' is a contribution to the question raised in DavidLHarden's answer: There are many more examples where the automorphism group is larger than the expected group, besides the two nice Mathieu cases and the trivial case where $S=F^\times$.

For this let $p$ be an odd prime, $F$ be the field with $p^2$ elements, $\mathbb F_p$ be the subfield of order $p$, and $S$ be the subgroup of $F^\times$ of order $2(p-1)$. So distinct elements $u,v\in F$ are joined if and only if $(u-v)^2\in\mathbb F_p$. Let $\omega\in F$ be of order $2(p-1)$. Identify the element $a+b\omega\in F$ with $(a,b)\in\mathbb F_p\times \mathbb F_p$. Then distinct pairs $(a,b)$, $(c,d)$ are joined if and only if $a=c$ or $b=d$. But any permutation on the first and another permutation on the second component preserves the graph structure, and so does the involution which switches the two components. Thus the automorphism group is at least as big as $(S_p\times S_p)\rtimes C_2$, where $S_p$ is the symmetric group on $p$ letters. (One easily shows that this is the full automorphism group.)

Of course one can generalize this example to get even more examples. One also gets examples where the automorphism group has more interesting composition factors, like $\text{PSL}_2(q)$ for certain prime powers $q$. Thus, I find the question interesting, and I'm not aware of results in the literature.