Suppose $A$ and $B=A^{-1}$ have all coefficients in $\mathbb N$. Then the same is true for the symmetric matrix $AA^t$ and its inverse $B^t B$. Positivity and symmetry of Since these mutually inverse matrices forces them to be are of the form $AA^t=I+a$ and $B^tB=I+b$ with $a$ and $b$ having coefficients in $\mathbb N$ and $I$ denoting the identity matrix. , we get $a=b=0$ by considering the product $(I+a)(I+b)=I+a+b+ab$.
The matrix $A$ is thus an orthogonal matrix with coefficients in $\mathbb N$, which implies that it is a permutation matrix.
