OP asked me to fill in the details of my comment, and in attempting to do so I realised that I claimed too much. However, a very similar argument proves a weaker result which is strong enough to support Max Alekseyev's answer (or Fedor Petrov's comment to it).
Theorem: if there is a counterexample, the smallest $n$ which is a counterexample is prime.
Suppose to the contrary that there is a counterexample and that the smallest counterexample is $n = ab$ with $a, b > 1$. We note that the non-vanishing of the determinant encodes the property that $k \to k \pi_k \bmod n$ is a permutation. The multiples of $a$ map to multiples of $a$; the values of $k$ for which $a | \pi_k$ also map to multiples of $a$. Therefore to preserve the bijective nature of the mapping, we require $\pi$ to map multiples of $a$ to multiples of $a$.
Considering the restriction of $\pi$ to multiples of $a$ we can define $\sigma_k = \frac{\pi_{ak}}{a}$ which must be a permutation of $[b]$. But then $k \to k \sigma_k$ is also a permutation of $[b]$, so we have a smaller counterexample, contradicting our assumption. $\blacksquare$