A general way to view these formulae is as a Parseval identity. If your measure were of the form $d\mu(x)=f(x)\,dx$, where $f\in L^2$, then you would have, for any set $E$ of finite positive measure,
$$
\mu(E)=\int_{\mathbb{R}^n}\mathbf{1}_Ef=\frac{1}{(2\pi)^n}\int_{\mathbb{R}^n} \hat{\mathbf{1}}_E\hat{\mathbf{f}}=\frac{1}{(2\pi)^n}\int_{\mathbb{R}^n} \hat{\mathbf{1}}_E\varphi_X.
$$
In the case $E$ is the unit ball, the Fourier transform is $\hat{\mathbf{1}}_E(\xi)=(2\pi)^{\frac{n}{2}}\lVert \xi\rVert^{-n/2}J_{n/2}(\lVert\xi\rVert)$, where $J$ is the Bessel function.
Of course, if $\mu$ is less regular, then the RHS does not need to converge absolutely. There are several ways to make sense of the formula nevertheless, namely:
- replace $\mathbf{1}_E$ with a test function, then pass to the limit;
- approximate $\mu$ with a more regular measure, then pass to a limit (as in the answer by Iosif Pinelis);
- replace the integral by its principal value, i. e., integrate over a box or ball of size $T$ centered about the origin and send $T$ to infinity.
The reason the last approach often works is because cutting out the high frequencies has a smoothening effect on both $\mathbf{1}_E$ and $f$, so it is essentially a combination of the first two. In particular, it always works when $\mu$ is supported away from the boundary of $E$; indeed, by writing $\mu=\mu\star\delta$ and exchangind integrals, it suffices to check the formula for delta-measures, in which case it is just the Fourier inversion formula at a point of smoothness of the original function.
If, on the other hand, $\mu$ is allowed to be supported on the boundary, then the v. p. integration may fail, but so does your formula for rectangles, as far as I can see. For example, in dimension 1, RHS is additive under the union $(a,b)\cup(b,c)$, while the LHS is not if $\mu(\{b\})\neq 0$.