**Update: corrected computational errors**

For what it's worth, here is a numerical plot of four approximations for the case $\mathbb Q(\sqrt{23})$, using coordinates $(x,y)$ to represent $x + y \sqrt{23}$. The plot covers a bit more than the fundamental domains for the quotient torus. The larger areas are points $p$ such that the norm of the difference with any of $5\times5$ array of grid points centered at the origin is greater than 1. The computation with a $13 \times 13$ grid shaves off the green areas, and the computation with a $21 \times 21$ grid shaves off the red. The dark purple areas are a computation taking into account lattice points $(n,m)$ with
$|n| \le 60$: three regions have split into four parts, the other two have shrunk and have not split. The black dots are the 46 fixed points on the torus for the Anosov map ${{24,115},{5,24}}$ corresponding to multiplication by a generator $24 + 5 \sqrt{23} $ of the group of units mod $\pm 1$.
**(An earlier version was incorrect: an error in a
formula made it appear that the regions were shrinking to 3 periodic points) **

It appears that $X$ is probably a Cantor set, or that at least it contains a Cantor set.

Since these sets are invariant by an Anosov diffeomorphism, I'm sure many things can be deduced using dynamics as well as number theory, but I thought I'd start by sharing these pictures.

Here is a plot of the minimum norm of the difference of points, in a small neighborhood of
the unit square, to an algebraic integer $n + m \sqrt{23}$ where $n \le 60$. The graph is
truncated at norm 1, so that the small plateaus correspond to the purple areas in the previous picture.

The globally minimum norm is the infimum of this function together with its images under the above Anosov map, whose eigenvalues are $47.979$ and $.02084$, since $60 > 47.97$. the infimum is a.e. 0, by ergodicity, as more and more long thin ravines are sliced through the landscape in two prevailing directions along the stable and unstable foliations of the Anosov diffeomorphism. The plateaus themselves will be stretched by a factor of ~50 in
one direction and a factor of .02 in the other.

For sufficiently small values of $a > 0$, the set of points whose
minimum norm of difference to a lattice point is $> a$ contains a Cantor set
(and I think it is a Cantor set). This can easily be seen by using a Markov partition consisting of rectangles whose edges are on leaves of the stable and unstable foliations, and translating the dynamics to symbolic dynamics (the sequence of rectangles that an orbit is in). Any closed set of sequences that does not contain a representative of $(0,0)$ on the
torus has a positive minimum value for norm.

Here's a similar plot, for the same field, of points whose minimum norm to the same set of algebraic integers is $> 1/2$:

This picture more strongly suggests that the intersection will be a Cantor set,
of significantly higher Hausdorff dimension. This is testable in principle using the dynamics. The Hausdorff dimension should be computable, at least numerically, using symbolic dynamics. For good values $b$ of the cutoff of norm, the closure
of the symbolic coordinates for correspondingset $X_b$ should be a subshift of finite type,
specified by a nonnegative matrix, and the Hausdorff dimension is a function of the
Perron-Frobenius eigenvalue compared to the eigenvalue for the symbolic dynamics on the full torus.

Here's a plot for the field $\mathbb Q(\sqrt{47})$, testing with all algebraic integers
$n + m \sqrt{47}$ where $|n| < 100$. Again this appears to indicate a Cantor set
of larger Hausdorff dimension than for $\mathbb Q(\sqrt {23})$:

Since there's a finite and known list of quadratic fields that are norm-Euclidean,
$\mathbb Q(\sqrt d)$ where $d = −11, −7, −3, −2, −1, 2, 3, 5, 6, 7, 11, 13, 17, 19, 21, 29, 33, 37, 41, 57, or 73,
(\emph{cf.}
[the Wikipedia article on Euclidean domains][5]), I suspect that in the typical
case, there are lots of points not close to being within norm 1 of a lattice point
(otherwise the criterion wouldn't be so successfull), so my guess is that most typically, the subset of the torus where minimum norm to a lattice
point is $> 1$ is a measure 0 Cantor set.

I hope someone with expertise I do not have will add an answer or comment (*added* since writing this, I'm glad to see that Franz Lemmermeyer has done so).