In "Smooth Homology Spheres and their Fundamental Groups" Kervaire proves that i) every 4-dimensional homology sphere bounds a contractible smooth manifold, and ii) for $d \geq 5$, every $d$-dimensional homology sphere bounds a contractible smooth manifold, after perhaps changing it by connect-sum with an exotic sphere.

Hence, let $\Sigma^d$ be any homology sphere of dimension $d \geq 4$, and modify it if necessary by connected-sum with a homotopy sphere so that it bounds a contractible manifold $\Delta^{d+1}$. Then
$$N := \Delta \cup_\Sigma \Delta$$
has an obvious (smooth) involution with fixed set $\Sigma$. Furthermore, this is easily seen to be a homotopy sphere (by Mayer--Vietoris and Seifert--van Kampen). It may not be diffeomorphic to $S^{d+1}$, but by changing $\Delta$ by connect-sum with an exotic sphere $Q$ we change $N$ by connect sum with $2Q$. So in those dimensions where the group $\Theta_{d+1}$ of exotic spheres has even order (which up to 20 is every $d+1$ except 13) we can always re-choose $\Delta$ so that $N$ is diffeomorphic to $S^{d+1}$.

Thus $\Sigma$ need not be a homotopy sphere in general