Is it true that $\det\big[\sin 2\pi\frac{(j-k)^2}p\big]_{1\le j,k\le p-1}=-\frac{p^{(p-1)/2}}{2^{p-1}}$ for each prime $p\equiv3\pmod4$? Question. Does the equality
$$\det\left[\sin 2\pi\frac{(j-k)^2}p\right]_{1\le j,k\le p-1}=-\frac{p^{(p-1)/2}}{2^{p-1}} $$
hold for every prime $p\equiv3\pmod4$?
I have checked the equality numerically for $p=3,7,11$. I conjecture that the equality holds for each prime $p\equiv3\pmod4$, but I don't know how to prove this.
Your comments are welcome!
 A: We will use the notation $e_p(t)=\exp\left(\frac{2\pi it}{p}\right)$. First, let us show that for any $1\leq m\leq \frac{p-1}{2}$ there is an eigenvector of your matrix $A_p$ with eigenvalue
$$
\lambda_m=\sqrt{p}\cos\frac{2\pi m^2}{p}.
$$
To do so, for any $m\in \mathbb Z$ denote by $v_m$ the vector from $\mathbb C^{p-1}$ with $k$-th coordinate $e_p(2mk)$ for all $k\leq p-1$. Computation of $j$-th coordinate of $A_pv_m$ gives
$$
(A_pv_m)_j=\sum_{k=1}^{p-1}e_p(2mk)\sin\frac{2\pi(j-k)^2}{p}=
$$
$$
=\frac{1}{2i}\sum_{k=1}^{p-1}e_p(2mk)(e_p((j-k)^2)-e_p(-(j-k)^2))=\frac{1}{2i}(S_{+}(m,j)-S_{-}(m,j)).
$$
Next, completing the sum, we get
$$
S_{+}(m,j)=\sum_{k=0}^{p-1}e_p(2mk+(j-k)^2)-e_p(j^2).
$$
Noticing that $2mk+(j-k)^2=(k+m-j)^2-m^2+2mj$, shifting the variables and using periodicity, we obtain
$$
S_+(m,j)=e_p(-m^2+2mj)\sum_{k=0}^{p-1}e_p(k^2)-e_p(j^2).
$$
It is known that the Gauss sum here equals $i\sqrt p$, thus $S_+(m,j)=i\sqrt pe_p(-m^2+2mj)-e_p(j^2)$. To compute $S_{-}(m,j)$, notice that $S_{-}(m,j)=\overline{S_+(-m,j)}=-i\sqrt{p}e_p(m^2+2mj)-e_p(-j^2)$. Combining these, we see that
$$
(A_pv_m)_j=\sqrt{p}\cos\frac{2\pi m^2}{p}e_p(2mj)-\sin\frac{2\pi j^2}{p}.
$$
Therefore, $A_pv_m=\lambda_m v_m-s$, where $s$ is a fixed vector whose $j$-th coordinate is $\sin\frac{2\pi j^2}{p}$. Noticing that $v_m\neq v_{-m}$ for $1\leq m\leq \frac{p-1}{2}$ and $\lambda_m=\lambda_{-m}$, we see that
$$
A_p(v_m-v_{-m})=\lambda_m (v_m-v_{-m}).
$$
Additional bonus here is the fact that coordinates of $v_m$ lie in $\mathbb Q(e_p(1))$. Let us take advantage of this. Denote $v_m-v_{-m}=w_m$. The cyclotomic field $\mathbb Q(e_p(1/4))=\mathbb Q(e_p(1),i)$ has an automorphism $\sigma$ that sends $e_p(1)$ to $e_p(-1)$ and $i$ to itself. One can define $\sigma$ via formula $\sigma e_p(1/4)=e_p((2p-1)/4)$ (which works, because $(2p-1,4p)=1$). This automorphism sends $\sin\frac{2\pi l}{p}$ to $\sin\frac{2\pi(p-1)l}{p}=-\sin\frac{2\pi l}{p}$ for integer values of $l$, so, for instance $A_p^\sigma=-A_p$. Therefore,
$$
A_p(\sigma^{-1}w_m)=\sigma^{-1}(A_p^\sigma w_m)=-\sigma^{-1}(A_pw_m)=-\lambda_m \sigma^{-1}w_m.
$$
Thus, all eigenvalues of $A_p$ are $\pm \lambda_m$ for $1\leq m\leq \frac{p-1}{2}$. This means that your identity is equivalent to
$$
(-1)^{(p-1)/2}\left(\prod_{m\leq \frac{p-1}{2}}\sqrt{p}\cos\frac{2\pi m^2}{p}\right)^2=-\frac{p^{(p-1)/2}}{2^{p-1}}.
$$
Finally, this identity is true if and only if
$$
\left(\prod_{m\leq \frac{p-1}{2}}2\cos\frac{2\pi m^2}{p}\right)^2=1.
$$
To establish this formula, notice that $p-m^2$ is not a square and it gives us the same cosine, so the identity is equivalent to
$$
\prod_{m\leq p-1}2\cos\frac{2\pi m}{p}=1.
$$
This last identity follows from the fact that $2\cos\frac{2\pi m}{p}=\frac{\sin\frac{4\pi m}{p}}{\sin\frac{2\pi m}{p}}$ and $m\mapsto 2m$ is a bijection on the set of invertible residues mod $p$.
EDIT: The proof is wrong, because I forgot that $\sigma$ also acts on $\lambda_m$. However, it seems that $-\lambda_m$ is still always an eigenvalue and then the rest of the proof works just fine.
EDIT 2: So, it turns out that the actual proof that $-\lambda_m$ is always an eigenvalue is quite difficult. I'll outline the proof here.
First of all, we notice that for any $k$ we have
$$
\sum_{j=0}^{p-1}\frac{1}{\cos\frac{2\pi j}{p}+\cos\frac{2\pi k}{p}}=0.
$$
To prove that, one can notice that
$$
\sum_{j=0}^{p-1}\frac{1}{z-\cos\frac{2\pi j}{p}}=\frac{T_p'(z)}{T_p(z)}=\frac{pU_{p-1}(z)}{T_p(z)},
$$
where $T_p$ and $U_{p-1}$ are Chebyshev polynomials. Also, $U_{p-1}(-\cos\frac{2\pi k}{p})=U_{p-1}(\cos\frac{\pi(p-2k)}{p})=0$, which concludes the proof of this observation. Now, the multisets $\{\cos\frac{2\pi m^2}{p}\}_{0\leq m\leq p-1}$ and $\{\cos\frac{2\pi m}{p}\}_{0\leq m\leq p-1}$ are the same, so we get, for instance, for all $m\not\equiv 0\pmod p$
$$
\sum_{j=0}^{p-1}\frac{1}{\lambda_j+\lambda_m}=0.
$$
From this we see, for example, that
$$
\sum_{j=1}^{p-1}\frac{\lambda_j-\lambda_0}{\lambda_j+\lambda_m}=p.
$$
This is a very useful identity for our proof. Next, we already showed that
$$
A_pv_m=\lambda_mv_m-s.
$$
One should also notice that $v_0=-v_1-\ldots-v_{p-1}$ and
$$
s=\frac{1}{p}\sum_{m=1}^{p-1}(\lambda_m-\lambda_0)v_m.
$$
Next, take
$$
w_m=\sum_{j=1}^{p-1}\frac{\lambda_j-\lambda_0}{\lambda_j+\lambda_m}v_j.
$$
Applying formulas for $A_pv_j$, we get
$$
A_pw_m=\sum_{j=1}^{p-1}\frac{\lambda_j-\lambda_0}{\lambda_j+\lambda_m}\lambda_jv_j-s\sum_{j=1}^{p-1}\frac{\lambda_j-\lambda_0}{\lambda_j+\lambda_m}.
$$
Due to our formula, the last coefficient is $p$, so
$$
A_pw_m=\sum_{j=1}^{p-1}\frac{\lambda_j-\lambda_0}{\lambda_j+\lambda_m}\lambda_jv_j-\sum_{j=1}^{p-1}(\lambda_j-\lambda_0)v_j.
$$
Therefore, $A_pw_m=-\lambda_mw_m$, which finally concludes the proof.
