Any finite order diffeomorphism of a closed manifold that fixes a point and the tangent space at that point is the identity (choose an invariant Riemannian metric and shoot geodesics away from that point). In particular $Diff_\partial(D^d)$ is torsion-free (extend the diffeomorphism to the sphere and use the above).

The fact that the kernel of $\eta^B$ for $B=\tau_{\ge k+1}BO$ (this is $BO\langle k\rangle$ in your notation, if I am not mistaken) is trivial follows from classical surgery, Kervaire--Milnor style: If $\Sigma$ is in the kernel of $\eta^B$, then it bounds a $k$-parallelisable $(2k+1)$-manifold. Doing surgery on in the interior of $W$, we see that $\Sigma$ bounds a $k$-connected $(2k+1)$-manifold $W'$. By Poincaré duality, $W'$ is contractible, so $W'\cong D^{2k+1}$ by the $h$-cobordism theorem, and thus $\Sigma\cong S^{2k}$.

As an aside: In odd dimensions $8k-1$, the kernel of the map $\Theta_{8k−1}/bP_{8k}\rightarrow\Omega^B_{8k-1}$ for $B=\tau_{4n}BO$ the $(4n-1)$-connected cover was determined only recently by Burklund--Hahn--Senger and Burklund--Senger, see arxiv.org/abs/1910.14116 and arxiv.org/abs/2007.05127 .

Here $\overline{\Sigma}$ is $\Sigma$ with the opposite orientation---the inverse in the group of homotopy spheres.

Just to clarify: no appeal to Kreck's work is necessary to answer the original question when exotic spheres are stably diffeomorphic. $\Sigma \sharp W_g\cong \Sigma'\sharp W_h$ implies $g=h$ by consulting the Euler characteristic. But then $W_g\cong \overline{\Sigma}\sharp \Sigma \sharp W_g\cong \overline{\Sigma}\sharp \Sigma'\sharp W_g$ and thus $\overline{\Sigma}\sharp \Sigma'\cong S^{2n}$ since the inertia group is trivial, so $\Sigma\cong\Sigma'$.

@JHM H is finite, so in particular arithmetic. The OP asked whether N(H) is arithmetic, not whether it is an arithmetic subgroup of G.