Here is an inequality one can in fact prove by the duality principle (Théorème 5 in Bombieri's Le grand crible, ignoring terms with $a\ne 0$):

$$\sum_{q\leq Q} \frac{\mu^2(q)}{q} \left(\frac{1}{N} \sum_{d|q} \mu(d) d \sum_{\substack{n\leq N\\d|n}} a_n\right)^2 \leq 2.$$

It is straightforward to use this inequality to prove that $\frac{1}{N/q} \sum_{n\leq N: q|n} a_n - \frac{1}{N} \sum_{n\leq N} a_n$ is small for most $q$ having a bounded number of small prime factors -- meaning $\sum_{q\in S_\epsilon} 1/q \gg_k (\log \log Q)^k$ for every $k\geq 1$. One can probabily push this argument up to $k = \sqrt{\log \log N}$ or so. 

Can one go farther, possibly by other means?