That set consists of all natural numbers. Consider the $n\times n$-matrix $A=(a_{ij})$ where all numbers on the diagonal, first row and first column are equal to 1, all the other entries are 0. Then the permanent of that matrix is $n$. Indeed, each non-zero summand in the definition of permanent should start with some $a_{1,i}$. Then the $i$-th factor of that product, $i > 1$, cannot be $a_{i,i}$. It must then be $a_{i,1}$. Hence all the other factors in that product are from the diagonal. Therefore exactly one non-zero summand in the permanent contains $a_{1,i}$, for every $i=1,...,n$. So the permanent contains exactly $n$ non-zero summands, and is equal to $n$.
4 of 4
added 9 characters in body
user6976