$\mathcal{Q}(x,\lambda)$ has positive relative density if and only if $\lambda\le 1$.
This follows from Weyl's Theorem on Uniform Distribution. (There is a nice concise proof   in Cassels' "Diophantine Approximation".) 



Weyl's Theorem:  Let $I\subset \mathbb{R}$ be an interval of length $\epsilon \le 1$. Let $S_N(I)$ be the set of all integers $q$ in the interval $[1,N]$ such that for some integer $p$, it holds that $xq-p\in I$. Then

 $$\frac{Card(S_N(I))}{N} \to \epsilon
 \text{ as } N\to\infty.$$



Here's a proof-sketch, using Weyl's Theorem, that if $\lambda > 1$ then $\mathcal{Q}(x,\lambda)$ has relative density zero:



Fix $\epsilon > 0$, and take $I$ (in Weyl's Theorem) to be the interval $(-\epsilon,\epsilon)$. Suppose $\lambda>1$.  Since $$|xq-p| < q^{1-\lambda},$$ we see that there is an integer $M$, depending only on $\epsilon$, such that $|xq-p| < \epsilon$ for all $q\ge M$. Therefore $$\mathcal{Q}(x,\lambda)\cap [M,N]\subset S_N(I)\cap [M,N].$$  It follows from Weyl's Theorem that the relative density of $\mathcal{Q}(x,\lambda)$ does not exceed $2\epsilon$. Since $\epsilon$ is arbitrary, the relative density of $\mathcal{Q}(x,\lambda)$ must be zero.


This can be proved in a more elementary but laborious way using the "Ostrowski Number System", which is explained in the Rockett and Szusz book on continued fractions.