Let $a_n$, $1\leq n\leq N$, be complex numbers, and set $S(\alpha)=\sum\limits_{n=1}^{N}a_ne(n\alpha)$, where $e(\alpha)=\exp(2i\pi\alpha)$. Then, Selberg's large sieve inequality says that $$\sum\limits_{q\leq Q}\sum\limits_{\substack{1\leq a\leq q\\ \gcd(a,q)=1}}\bigg|S\left(\frac{a}{q}\right)\bigg|^2\leq(N+Q^2)\sum\limits_{n=1}^{N}|a_n|^2.$$ I have heard that there are possible refinements of the large sieve or some special sequences $a_n$, for which the factor $N+Q^2$ could be replaced by $N+cQ^2$, for a constant $0<c<1$. But I couldn't find any relevant results except a Rutger's PhD thesis. So, any reference to such results or collection of such results would be highly appreciated. Generally, is it possible to reduce that factor in a similar way for any sequence $a_n$ by introducing suitable weights? Does there exist any work on this?
2 Answers
For general sequences $(a_n)$, the only further improvement that can be made is replacing $N+Q^2$ with $N+Q^2-1$. This was proved by Selberg, and it has a nice treatment in Chapter 9 of Opera de Cribro (Friedlander and Iwaniec).
There is a nice example of the sort of refinement that you are looking for when you consider the multiplicative large sieve (where the frequencies are primitive Dirichlet characters instead of complex exponentials). In Theorem 9.11 of Opera de Cribro, it is proved that if $a_n = 0$ when the least prime dividing $n$ is $\leq Q$, then
$\displaystyle \sum_{q\leq Q}\log\Big(\frac{Q}{q}\Big)\sideset{}{^*}\sum_{\chi\pmod{q}}\Big|\sum_{M<n\leq M+N}a_n \chi(n)\Big|^2 \leq (N+Q^2-1)\sum_{M<n\leq M+N}|a_n|^2,$
where $\sum^*$ is a sum over primitive Dirichlet characters. The $\log(Q/q)$ weight on the LHS is decisive in certain settings, going far beyond a savings in the constant in front of $Q^2$. For example, this sensitivity of the large sieve inequality to sequences supported on large primes (or more generally, to sequences supported on integers with only large prime factors) is crucial for the sort of "log-free" zero density estimates for Dirichlet $L$-functions that are used to prove Linnik's bound on the least prime in an arithmetic progression. See Gallagher's Inventiones paper "A large sieve density estimate near $\sigma=1$" for further discussion.
(You can think of this weighted large sieve inequality as a synthesis of the classical large sieve inequality and Selberg's sieve.)
Though a bit late -- In general, look at Theorem 2.1 of https://ramare-olivier.github.io/Maths/Eigenvalues-JTNB.pdf This improves a bit on the c in (N+cQ^2). For a generic sifted set, look at Theorem 5 of https://ramare-olivier.github.io/Maths/Ramare_Ruzsa.pdf Or same in my blue book Theorem 5.2 (Arithmetical Aspect of the Large Sieve Inequality). For the primes, it gives Theorem 5.3:
If $(u_n)_{n\le N}$ is such that $u_n$ vanishes as soon as $n$ has a prime factor less than $\sqrt{N}$, then \begin{equation*} \sum_{q\le Q_0}\sum_{a\mod^* q} \biggl|\sum_{n}u_n e(na/q)\biggr|^2 \le 7\,\frac{N\log Q_0}{\log N}\sum_{n}|u_n|^2 \end{equation*} for any $Q_0\le \sqrt{N}$ and provided $N\ge100$.
Maybe off-topic:
If your sequence is supported on a polynomial sequence, look at "The large sieve inequality for real quadratic polynomial amplitudes" by G.Prakash and S.Ramana, and if you are into algebraic varieties, look at "Large sieve inequalities for algebraic trace functions" by Xi, Ping, appendix by Fouvry, Kowalski and Michel.
I hope that helps! Best, Olivier