I believe the answer is negative and a counterexample might be strongly regular graph.
On 35 vertices there are 3854 SRGs, all cospectral.
SRG need not be Cayley.
Since Paley graphs of prime order are SRG circulants, this approach may settle the second question too.