There are only finitely many such matrices. Let us consder the circular matrix say $A$, with the first row $(i,j,k,m)$. Then $det(A)=FGH$, where
$$
F=i-j+k-m,\; G=i+j+k+m,\;H=(i-k)^2+(j-m)^2
$$
We need to have $det(A)=\pm 1$. Thus, we need to play with the following system of equations
\begin{align*}
 (A) \quad &F=G=H=1,\\
 (B) \quad &F=G=-1, H=1,\\
 (C) \quad &F=-1, G=1, H=1,\\
 (D) \quad &F=1, G=-1, H=1.
 \end{align*}
Is is a simple excercise to prove that each of the above systems has only finitely many solotions.