Non-singular matrix with restricted entries

Question

Given a set $S$ of integers with $1 \not\in S$, let us consider the set $\mathcal{M}$ of all the symmetric matrices $M$, such that:

(i) All the diagonal entries of $M$ are equal to $1$.

(ii) All the off-diagonal entries of $M$ are from $S$.

Obviously, if $S$ only consists of numbers divisible by a prime number $p$, then a $M \in \mathcal{M}$ is always non-singular. This can be seen by either analyzing its rank over $\mathbb{F}_p$, or just expanding its determinant.

Now, the question is, is it true that every $S$ such that all satisfiable $M$ are non-singular must be a subset of $\{\cdots, -2p, -p, 0, p, 2p, \cdots\}$ for some prime $p$? I feel that this must have been studied in the literature but was not able to find it after extensive search. A natural thing to try first is $S=\{k, k+1\}$ for $k \ge 2$, one can actually construct the following singular $2k \times 2k$ (symmetric) matrix: \begin{bmatrix} (k+1)J_k-kI_k & kJ_k\\ kJ_k & (k+1)J_k-kI_k\\ \end{bmatrix} It is singular because the sum of the first $k$ rows is equal to the sum of the last $k$ rows.

Answer 1

This is perhaps a minor observation.

If $S$ has every admissible symmetric matrix being non-singular, then $\{0,-1\}\not\subseteq S$.

If that were not the case then the singular $4\times 4$ matrix $\begin{pmatrix} I&-I\\-I& I\end{pmatrix}$, where $I$ is the $2\times 2$ identity matrix, would be admissible.

Added Later

More generally,

If $S$ has every admissible symmetric matrix being non-singular, then $\{n,2n^2-1\}\not\subseteq S$ for any integer $n\ne\pm 1$.

The previous case is recovered when $n=0$; however, here the counterexample follows from the singular $3\times 3$ symmetric matrix $$\begin{pmatrix} 1&n&n\\n&1&2n^2-1\\n&2n^2-1&1\end{pmatrix}\,.$$