A special case is the following:
Pick:
An integer $n$ that is a square;
$$ H =F^{*} D F $$
a matrix with $n$ lines and $n$ columns
where
$D$ is a diagonal square matrix with $n$ lines and with integer coefficients $F$ is the Fourier matrix, with $n$ lines defined by
$$ F = (1/\sqrt{n}) (s^{(i-1)(j-1)}) $$
where
$$ s = e^{-2 \pi i/n} $$
$*$ means conjugate-transpose.
You get
$H$ is hermitian with entries algebraic integers.
$H$ is also a circulant matrix.
Pick now:
$$ U =F^{*} $$
so that
$$ Q(H) \subseteq Q(U) = Q(s) $$
while
$$ Q(U,D) = Q(F^{*},D) = Q(s). $$
Observe that $$ Q(s) $$ is the classic extension of $Q$ containing the $n$-th roots of unity so that it has degree
$$ \varphi(n) $$
over $Q$, where $\varphi$ is the Euler totient's function.
Thus,
The extension $Q(U,D)$ over $Q(H)$ has degree $d$ bounded above by $\varphi(n)$\varphi(n).$
Observe that this degree $d$ is substantially slower than $n !$ since
$$ d \leq \varphi(n) < n. $$
