Orangeskid's guess is correct: a more general fact holds that the binomial transform preserves Hankel determinants.
For a matrix $(a_{ij})$ (it is convenient to enumerate rows and columns from 0,not from 1) denote $$b_{ij}=\sum_{k, s}{i\choose k}{j\choose s}a_{ks}.$$ This matrix transform corresponds to a left and right multiplicaton by unitriangular matrices, thus it preserves determinant. Now if $a_{ij}=f(i+j)$ is a Hankel matrix, then $$b_{ij}=\sum_t f(t)\sum_{k+s=t} {i\choose k}{j\choose s}=\sum_tf(t){i+j\choose t}$$ is a Hankel matrix corresponding to the binomial transform of $f$.
It remains to recall that $n! =\sum_k {n\choose k} d_k$ (combinatorially ${n\choose k} d_k$ counts the number of permutations of $\{1,\ldots, n\}$ with exactly $n-k$ fixed points, thus this formula), that means that the sequence of factorials is the binomial transform of the sequence of dearrangements.
