I believe the answer is negative and a counterexample might be strongly regular graph.

On [35 vertices](https://www.win.tue.nl/~aeb/graphs/srg/srgtab1-50.html) there are 3854 SRGs, all cospectral.

SRG need not be Cayley.

---

Since Paley graphs of prime order are SRG circulants, this approach may settle the second question too.