On the basis of my computation, I have the following conjecture involving the secant function.
Conjecture. Let $p$ be an odd prime and define $$S_p:=\det\left[\sec2\pi\frac{jk}p\right]_{0\le j,k\le (p-1)/2}.$$ If $p\equiv 1\pmod4$, then $S_p=0$. If $p\equiv3\pmod4$, then $$s_p=\frac{(\frac 2p)S_p}{2^{(p-3)/2}p^{(p+1)/4}}$$ is a positive odd integer.
Remark. Via Mathematica I find that $s_3=s_7=s_{11}=1$ and $s_{19}=19$.
Let $p$ be an odd prime. For $a=1,\ldots,p-1$ let $\pi_a$ be the permutation on $\{0,1,\ldots,(p-1)/2\}$ such that $\pi_a(k)$ is the unique $r\in\{0,\ldots,(p-1)/2\}$ with $ak$ congruent to $r$ or $-r$ modulo $p$. By a result of H. Pan in the paper arXiv:0601026, the sign of $\pi_a$ is $(\frac ap)^{(p+1)/2}$. If $p\equiv3\pmod4$, then by applying the automorphism $\sigma_a$ in the Galois group of $\mathbb Q(e^{2\pi i/p})$ with $\sigma(e^{2\pi i/p})=e^{2\pi ia/p}$, we see that $$\sigma_a(S_p)=\det\left[\sec 2\pi\frac{ajk}p\right]_{0\le j,k\le(p-1)/2}=\left(\frac ap\right)^{(p+1)/2}S_p$$ with the help of Pan's result. Thus, when $p\equiv3\pmod4$ we have $\sigma_a(S_p)=S_p$ for all $a=1,\ldots,p-1$, and hence $S_p$ is rational. If $p\equiv1\pmod 4$, then for each $a=1,\ldots,p-1$ we have $$\sigma_a(\sqrt p)=\sum_{x=0}^{p-1}\sigma_a(e^{2\pi ix^2/p})=\sum_{x=0}^{p-1}e^{2\pi iax^2/p}=\left(\frac ap\right)\sqrt p.$$ Thus, $\sigma_a(S_p/\sqrt p)=S_p/\sqrt p$ for all $a=1,\ldots,p-1$, and hence $S_p/\sqrt p\in\mathbb Q$.
Similarly, I can show that for any odd prime $p$ the numbers $$\frac1{2^{(p-1)/2}p^{(p-3)/4}}\det\left[\sec2\pi \frac{jk}p\right]_{1\le j,k\le(p-1)/2}$$ and $$\frac1{2^{(p-1)/2}p^{(p-5)/4}}\det\left[\csc2\pi \frac{jk}p\right]_{1\le j,k\le(p-1)/2}$$ are rational numbers. I conjecture that these two numbers are always integers.
Your comments are welcome!
