Given a set $S$ of integers with $1 \not\in S$, let us consider the set $\mathcal{M}$ of all the symmetric matrices $M$, such that:
(i) All the diagonal entries of $M$ are equal to $1$.
(ii) All the off-diagonal entries of $M$ are from $S$.
Obviously, if $S$ only consists of numbers divisible by a prime number $p$, then a $M \in \mathcal{M}$ is always non-singular. This can be seen by either analyzing its rank over $\mathbb{F}_p$, or just expanding its determinant.
Now, the question is, is it true that every $S$ such that all satisfiable $M$ are non-singular must be a subset of $\{\cdots, -2p, -p, 0, p, 2p, \cdots\}$ for some prime $p$? I feel that this must have been studied in the literature but was not able to find it after extensive search. A natural thing to try first is $S=\{k, k+1\}$ for $k \ge 2$, one can actually construct the following singular $2k \times 2k$ (symmetric) matrix: \begin{bmatrix} (k+1)J_k-kI_k & kJ_k\\ kJ_k & (k+1)J_k-kI_k\\ \end{bmatrix} It is singular because the sum of the first $k$ rows is equal to the sum of the last $k$ rows.