I have found a solution myself, at least in the case when $a$ and $b$ are partitions. 

The determinant can be written as
$$ D=\det((x_i+y_j)!)=\det\left( \int z^{x_i+y_j}e^{-z}dz\right)$$
We resort to the Andreief identity,
$$ \int dz \det(f_{i}(z_{j}))\det(g_{i}(z_{j}))=N!
\det\left(\int f_{i}(z)g_{j}(z)dz\right).$$
This is usually used from left to right, but I used in reverse to write
$$D=\frac{1}{N!}\int dz \det(z_{j}^{x_j})\det(z_{j}^{y_i})\prod_{i=1}^Ne^{-z_i}$$
Now, if $x_i=a_i-i+N$ and $y_i=b_i-i+N$ and if $a\vdash n$ and $b\vdash m$ are partitions, then $\det(z_{j}^{x_j})\det(z_{j}^{y_i})=(V(z))^2s_a(z)s_b(z)$, where $V$ is the Vandermonde and $s$ are the Schur functions. Then
$$D=\frac{1}{N!}\int dz (V(z))^2s_a(z)s_b(z)\prod_{i=1}^Ne^{-z_i}$$

The Littlewood-Richardson coefficients are defined by $s_as_b=\sum_{\rho\vdash n+m} c^\rho_{ab}s_\rho$. Using them we have
$$D=\frac{1}{N!}\sum_{\rho\vdash n+m} c^\rho_{ab}\int dz (V(z))^2s_\rho(z)\prod_{i=1}^Ne^{-z_i}.$$
This is a integral of the Selberg type, and the result is known:
$$\int dz (V(z))^2s_\rho(z)\prod_{i=1}^Ne^{-z_i}=N!\frac{d_\rho}{(n+m)!}\prod_{j=1}^N (\rho_i+N-i)!^2 \quad (\ell(\rho)\le N),$$ where $d_\rho$ is the dimension of the irreducible representation of the permutation group labeled by $\rho$.

Therefore,
$$D=\frac{1}{(n+m)!}\sum_{\substack{\rho\vdash n+m\\\ell(\rho)\le N}} d_\rho c^\rho_{ab}\prod_{j=1}^N (\rho_i+N-i)!^2.$$

Curiously, there are no Vandermonds in this solution.