A new formula for the class number of the quadratic field $\mathbb Q(\sqrt{(-1)^{(p-1)/2}p})$? I have the following conjecture involving a possible new formula for the class number of the quadratic field $\mathbb Q(\sqrt{(-1)^{(p-1)/2}p})$ with $p$ an odd prime.
Conjecture. Let $p$ be an odd prime and let $p^*=(-1)^{(p-1)/2}p$. Then the class number $h(p^*)$ of the quadratic field $\mathbb Q(\sqrt{p^*})$ coincides with the number
$$D(p):=\frac{(\frac{-2}p)}{2^{(p-3)/2}p^{(p-5)/4}}\det\left[\cot\pi\frac{jk}p\right]_{1\le j,k\le (p-1)/2},$$
where $(\frac{\cdot}p)$ is the Legendre symbol.
This is Conjecture 5.1 in my preprint arXiv:1901.04837. I have checked it for all odd primes $p<29$. Note that $h(p^*)=1$ for each odd prime $p<23$, and $h(-23)=3$.
Here I invite some of you to check this conjecture further. My computer cannot check it even for $p=29$.
Edit: F. Brunault computered $D(p)$ for $p=29,31,37,41,43,47$ and noted that the conjecture is false. However, I believe that $h(p^*)\mid D(p)$ for any odd prime $p$.
Now I explain why $D(p)\in\mathbb Q$ by Galois theory. The Galois group $\text{Gal}(\mathbb Q(e^{2\pi i/p})/\mathbb Q)$ consisits of those authormorphisms $\sigma_a$ $(1\le a\le p-1)$ with $\sigma_a(e^{2\pi i/p})=e^{2\pi ia/p}$. By Gauss' Lemma,
$$\left(\frac ap\right)=(-1)^{|\{1\le j\le(p-1)/2:\ \{aj/p\}>1/2\}|}.$$
For $j=1,\ldots,(p-1)/2$ let $\pi_a(j)$ be the unique $r\in\{1,\ldots,(p-1)/2\}$ with $aj\equiv \pm r\pmod p$. Then, for $D=\det[\cot\pi\frac{jk}p]_{1\le j,k\le(p-1)/2}$, using $p$th roots of unity we get
\begin{align}\sigma_a\left(\frac D{i^{(p-1)/2}}\right)=&\frac1{i^{(p-1)/2}}\det\left[\cot\pi\frac{ajk}p\right]_{1\le j,k\le(p-1)/2}
\\=&\frac{(\frac ap)}{i^{(p-1)/2}}\det\left[\cot\pi\frac{\pi_a(j)k}p\right]_{1\le j,k\le(p-1)/2}
\\=&\left(\frac ap\right)\left(\frac ap\right)^{(p+1)/2}\frac{D}{i^{(p-1)/2}}=\left(\frac ap\right)^{(p-1)/2}\frac{D}{i^{(p-1)/2}}\end{align}
since $\text{sign}(\pi_a)=(\frac ap)^{(p+1)/2}$ as pointed by Pan in arXiv:0601026. Thus, if $p\equiv1\pmod4$ then $\sigma_a(D)=D$ for all $a=1,\ldots,p-1$ and hence $D\in\mathbb Q$. When $p\equiv3\pmod4$, we have
$$\sigma_a\left(\frac D{\sqrt{p}}\right)=\left(\frac ap\right)\frac D{i^{(p-1)/2}}\cdot\frac1{\sigma_a(\sqrt{p^*})}.$$
Using Gauss' sums we see that
$$\sigma_a(\sqrt{p^*})=\sum_{x=0}^{p-1}e^{2\pi iax^2/p}=\left(\frac ap\right)\sqrt{p^*}.$$
Therefore $\sigma_a(D/\sqrt p)=D/\sqrt p$ for all $a=1,\ldots,p-1$, and hence 
$D/\sqrt p\in\mathbb Q$.
 A: If by
"the class number $h(p^*)$ of the quadratic field $\mathbb{Q}(\sqrt{p^*})$"   
you mean
"the minus class number $h^{-}$ of $\mathbf{Q}(\zeta_p)$"
and if by
" a possible new formula for the class number"  
you mean
"an elementary formula for $h^{-}$ known to Kummer (obtained by considering the ratio of the zeta functions of $\mathbf{Q}(\zeta_p)$ and the totally real subfield $\mathbf{Q}(\zeta_p)^{+}$ at s = 1)"
Then you are correct.
A: The conjecture is not true, as some examples show.
Let $D(p)$ denote your number. For primes $p \equiv 1 \textrm{ mod } 4$, we have $D(29)=8$, $D(37)=37$, $D(41)=121$ while $h(29)=h(37)=h(41)=1$.
For primes $p \equiv 3 \textrm{ mod } 4$, we have $D(23)=3$, $D(31)=9$, $D(43)=211$, $D(47)=695$ while $h(23)=h(31)=3$, $h(43)=1$, $h(47)=5$.
The following Pari/GP script calculates $D(p)$:
D(p)=kronecker(-2,p)*matdet(matrix((p-1)/2,(p-1)/2,j,k,cotan(Pi*j*k/p)))/(2^((p-3)/2)*p^((p-5)/4))

(This is a floating-point computation, but it is easy to be rigorous by replacing the entries of the matrix by algebraic numbers). It seems that $D(p)$ grows fast.
EDIT. For primes $p \equiv 1 \textrm{ mod } 4$, divisibility does not hold, for example $h(229)=h(257)=3$ but $D(229) \equiv D(257) \equiv 1 \textrm{ mod } 3$.
For primes $p \equiv 3 \textrm{ mod } 4$, one should be able to relate $D(p)$ with the minus class number $h(\mathbb{Q}(\zeta_p))^-$ as alluded to by user134696 using the analytic class number formula and the fact that $D(p)$ looks like a group determinant on $(\mathbb{Z}/p\mathbb{Z})^\times$, so should be a product over Dirichlet characters mod $p$.
The point is that Dirichlet's class number formula relates $h(p^*)$ with $L(\chi_p,1)$ where $\chi_p$ is the Legendre symbol, but $\chi_p$ is even in the case $p \equiv 1 \textrm{ mod } 4$ so it should have nothing to do with $h(\mathbb{Q}(\zeta_p))^-$ in this case.
