Consider a Dirichlet series $\sum_n a_n n^{-s}$ with desirable analytic properties (e.g., analytic extension to $\Re s>0$); one example would be $a_n=\mu(n)$. Say we want to estimate $\sum_{n\leq x} a_n$, and that we are given that the Riemann hypothesis has been verified for all non-trivial zeros $\rho$ with $|\Im \rho|\leq T$.
Then we want to choose a smoothing function $\eta:[0,\infty)\to \mathbb{R}$ close to $1_{[0,1]}$ so that we can estimate $\sum_{n} a_n \eta(n/x)$ by analytic means and then add the error term $\sum_{n\leq x} a_n |\eta(n/x)-1| + \sum_{n>x} a_n |\eta(n/x)|$, which, if $a_n$ is bounded (in $L^\infty$ or even $L^1$ norm), should be at most about $x \int_0^\infty |\eta(t)-1_{[0,1]}(t)| dt$. In the estimation of $\sum_{n\leq x} a_n \eta(n/x)$, the main term will be very roughly of the form $$\kappa x \int_T^\infty |M\eta(1+it)| dt,$$ where $\kappa$ is a constant depending on the series and $M\eta$ is the Mellin transform of $\eta$.
The question then is how to minimize $$I = \int_0^\infty |\eta(t)-1_{[0,1]}(t)| dt + \kappa \int_T^\infty |M\eta(1+it)| dt,$$ and what that minimum shall be.
Can one get the best $\eta$ and the best $I$? Or at least get the minimum of $I$ within a constant factor?
The problem of choosing $\eta$ has of course been considered in the literature (for $\sum_n a_n n^{-s} = \zeta(s)$: I am aware of Büthe and Faber-Kadiri) but not in ways that I find completely satisfactory. Let us consider one plausible candidate for $\eta$.
Let $\eta$ be given by $$\eta(t) = \frac{1}{2} \left(1 - \mathrm{erf}\left(\frac{\log t}{\delta/T}\right)\right),$$ where we can choose $\delta$. This choice has the advantage that the Mellin transform $M\eta$ of $\eta$ is simply $$M\eta(s) = e^{\left(\frac{\delta s}{2 T}\right)^2}.$$ If we set $\delta$ of size close $\sqrt{8 \log T}$, we obtain a bound on $I$ with leading-order term $$\frac{\sqrt{(8/\pi) \log T}}{T}.$$
Can one do better than this? Can one do better than $O(\sqrt{\log T}/T)$? It is not hard to show that one cannot do better than $O(1/T)$ (see the answer to Fourier optimization problem related to the Prime Number Theorem).