I believe the answer is negative and a counterexample might be strongly regular graph.
On 35 vertices there are 3854 SRGs, all cospectral.
SRG need not be Cayley.
joro
- 25.4k
- 10
- 66
- 121