Here is a (I think) mathematically correct, but clearly morally wrong, answer via extreme overkill.
Upon multiplying by $i$, we may work instead with skew-Hermitian matrices, i.e., with $\mathfrak u(n)$. The usual proof of the Weyl integration formula on $U(n)$ works just as well for $\mathfrak u(n)$, and gives that
$$
\tag{W}\label{W}
\int_{\mathfrak u(n)} f(X)\mathrm dX = \int_{\mathfrak t} f(Y)\lvert D(Y)\rvert^{1/2}\mathrm dY
$$
for all continuous, compactly supported, $U(n)$-fixed functions $f$ on $\mathfrak u(n)$, where $\mathfrak t$ is the subspace of diagonal matrices in $\mathfrak u(n)$ and $D(Y) = \prod_{\alpha \in \Phi(\mathfrak u(n), \mathfrak t)} \alpha(Y)$ equals the product of pairs of differences of eigenvalues of $Y$. In particular, the singular set $S$ is the $0$ set of $D$, hence, as @StevenLandsburg suggested, is closed.
Now I'll use dominated convergence freely. To show that $S$ has measure $0$, it suffices to show that its intersection with every compact subset $K$ of $\mathfrak u(n)$ has measure $0$. Upon replacing such a set $K$ by its $U(n)$-orbit, we may, and do, assume that $K$ is $U(n)$-stable.
Now there is a sequence $(f_k)_{k = 1}^\infty$ of continuous, compactly supported functions, all pointwise between $0$ and $1$, such that $f_k$ tends pointwise to the indicator function of $S \cap K$.
Since $S$, hence $S \cap K$, is stable under $U(n)$-conjugacy, upon replacing each $f_k$ by the average of its $U(n)$-orbits, we may, and do, assume that the sequence consists of continuous class functions. Then, upon applying \eqref{W} to each such function and taking the limit, we see that
$$
\operatorname{meas}(S \cap K) = \int_{S \cap K} \mathrm dX = \int_{\mathfrak t \cap S \cap K} \lvert D(Y)\rvert^{1/2}\mathrm dY = 0.
$$