Let $C$ be a smooth projective curve of genus $g$ with an involution $\iota: C \to C$. We have the quotient map $\pi: C \to C/\iota$, with $C/\iota$ a smooth curve of genus $h$. The pullback map $\pi^{*}: \text{Jac}(C/\iota) \to \text{Jac}(C)$ in this case is injective, and we define the Prym variety as the cokernel:
$$0 \to \text{Jac}(C/\iota) \to \text{Jac}(C) \to \text{Prym}(\pi) \to 0$$
Note that $\text{Prym}(\pi)$ is an Abelian variety of genus $g-h$. Now consider the action by $\iota^{*}$ on $\text{Jac}(C)$. The induced action on $\text{Jac}(C/\iota)$ is trivial, i.e. line bundles pulled back from the quotient are $\iota$-invariant. I am nearly certain that the induced action on $\text{Prym}(\pi)$ is by $\pm 1$ and therefore has $2^{2g-2h}$ fixed points. But I am struggling to prove this. Anyone have any tips? I'm unsure how to prove that a particular involution on an Abelian variety is simply the standard $\pm 1$ action.
(I think an equivalent statement is that the only deformations of invariant degree $0$ line bundles on $C$ come from deformations on $C/\iota$. But there must be a more direct way to argue.)