$\mathcal{Q}(x,\lambda)$ has positive relative density if and only if $\lambda\le 1$. This follows from Weyl's Theorem on Uniform Distribution. (There is a nice concise proof in Cassels' "Diophantine Approximation".)
Weyl's Theorem: Let $I\subset \mathbb{R}$ be an interval of length $\epsilon \le 1$. Let $S_N(I)$ be the set of all integers $q$ in the interval $[1,N]$ such that for some integer $p$, it holds that $xq-p\in I$. Then
$$\frac{Card(S_N(I))}{N} \to \epsilon \text{ as } N\to\infty.$$
Here's a proof-sketch, using Weyl's Theorem, that if $\lambda > 1$ then $\mathcal{Q}(x,\lambda)$ has relative density zero:
Fix $\epsilon > 0$, and take $I$ (in Weyl's Theorem) to be the interval $(-\epsilon,\epsilon)$. Suppose $\lambda>1$. Let $q\in \mathcal{Q}(x,\lambda)$; so for some $p\in \mathbb{Z}$, $$|xq-p| < q^{1-\lambda}.$$
There is an integer $M$, depending only on $\epsilon$, such that $|xq-p| < \epsilon$ whenever $p$ and $q$ satisfy the above inequality and $q\ge M$. Therefore $$\mathcal{Q}(x,\lambda)\cap [M,N]\subset S_N(I)\cap [M,N].$$ It follows from Weyl's Theorem that the relative density of $\mathcal{Q}(x,\lambda)$ does not exceed $2\epsilon$. Since $\epsilon$ is arbitrary, the relative density of $\mathcal{Q}(x,\lambda)$ must be zero.
This can be proved in a more elementary but laborious way using the "Ostrowski Number System", which is explained in the Rockett and Szusz book on continued fractions.