Consider $A$ the order $n$ adjacency matrix which has 1 in the $i,j$th entry if there is an edge in the bipartite graph $G$ from top vertex $i$ to bottom vertex $j$, 0 if not. Then the permanent of $A$ counts the number of perfect matchings in $G$. One now sees that any such bipartite graph missing $k\lt n$ edges is a multiple of $(n-k)!$, which has some significance when $k$ is small. In particular when $k=2$, the number of matchings is a multiple of $(n-2)!$. So the feasible values near $n!$ are indeed few in number.
Elsewhere (Number of unique determinants for an NxN (0,1)-matrix) I talk about the determinant spectrum problem for order $n$ $0-1$ matrices; a similar problem for permanents can be and probably has been asked, for which I would like to see references.
Gerhard "Ask Me About Multilinear Functions" Paseman, 2016.05.18.