I believe the answer is negative and a counterexample might be strongly regular graph.

On [35 vertices](https://www.win.tue.nl/~aeb/graphs/srg/srgtab1-50.html) there are 3854 SRGs, all cospectral.

SRG need not be Cayley.