Optimizing a smoothing function with the Prime Number Theorem in mind Let $f:[0,\infty)\to \mathbb{R}$ be a function with $f(x)=1$ for $0\leq x\leq 1$. Write $Mf$ for the Mellin transform of $f$. Let $c>0$, $T>10^6$ be constants. We are interested in minimizing the following quantity:
$$|f-1_{[0,1]}|_1 + c \int_T^\infty |(Mf)'(1 + it)| \cdot(t-T)  dt.$$
What choice of $f$ makes it minimal?
The applications to estimating $\psi(x)$, $M(x)$, etc., should be more or less clear. (See Best smoothing for the Prime Number Theorem?)

A reasonable choice would be of the following form: choose $y>1$ close to $1$, and $k\geq 1$; write $\Delta_y^{\cdot} g = g(x y)-g(x)$; define $f = \frac{(\log y)^{-k}}{k!} (\Delta_{1/y}^\cdot)^k g,$ where $g(x) = \log^k(1/x)$ for $0<x\leq 1$ and $g(x)=0$ for $x>1$. By an appropriate choice of $k$ (not too small) and $y$ (proportional to a negative fractional power of $T$), one can get good results, but I do not know how far they are from optimal.
 A: This is not an answer, but rather a working-out of the choice of $f$ I mentioned in the question.
Let $f = \frac{(\log y)^{-k}}{k!} (\Delta_{1/y}^\cdot)^k g,$ where $g(x) = \log^k(1/x)$ for $0<x\leq 1$ and $g(x)=0$ for $x>1$. It is easy to see that
$f(x)=1$ for $0\leq x\leq 1$ and $f(x)=0$ for $x\geq y^k$. Hence
$$|f-1_{[0,1]}|_1 = \sum_{j=0}^{k-1} \int_{y^j}^{y^{j+1}} f(x) dx = \dotsc$$
Wait, the long answer I had lovingly crafted just got erased when I closed a window. (I had been working off-line.) The final conclusion was that the quantity we wish to minimize ends up being roughly $(k/2) (2/T)^{(k-1)/k}$, that we are best off taking $k = (\log T)/2$, and that the bound then is about
$$\frac{e \log T}{T}.$$
Can one do much better?
(If one drops the condition $f(x)=1$, then of course one can do better, by using $f(y^{k/2} x)$ instead of $f$. Then one saves a constant of at least $2$. I can work this out in lieu of my vanished answer, if people are interested.)
A: Let me redo things with $f(x) = \frac{\delta^{-k}}{k!} \Delta_\delta^k g(x-k\delta)$, where $g(x) = (1-x)^k$ for $x\leq 1$ and $g(x)=0$ for $x>1$, and $\Delta h = h(y+\delta) - h(y)$. It is almost identical to the previous choice (with the multiplicative difference $\Delta_y^\cdot$) for $y = 1 + \delta$, $\delta$ small, and the initial analysis is slightly easier. Let me also minimize the related quantity
$$S_0 = |f(x)-1_{[0,1]}| + c \int_T^\infty |Mf(1+it)| dt$$
instead of the quantity of the statement: first, this is what actually needs to be minimized if one proves number-theoretical bounds in the traditional way (though I do need the quantity in the statement, personally); second, if we work with the quantity in the statement, there is a technical glitch that appears for this choice of $f$ and not for $f(x) = \frac{\delta^{-k}}{k!} (\Delta^\cdot_{1/y})^k \log^k(1/x)$. (If the solution to that glitch does not become apparent, then the latter choice of $f$ would indeed be preferrable.)
I will give a variant later with $f(x) = \frac{\delta^{-k}}{k!} \Delta_\delta^k g(y-k\delta/2)$ (valid if we drop the condition that $f(x)=1$ for all $0\leq x\leq 1$); it gives better results.
It should be clear that $f(x)=1$ for $0\leq x\leq 1$ and  $f(x)=0$ for $x\geq y + k \delta$. We can write $\Delta_\delta^k g(y-k\delta)$ as the convolution of $g_k$ with $ 1_{[0,\delta]}^{* k}= 1_{[0,\delta]}\ast \dotsc \ast 1_{[0,\delta]}$ ($k$ times), where $g_k(x) = g^{(k)}(x) = k!$ for $x\leq 1$ and $g_k(x) = 0$ for $x>1$. (Thanks to Noam Elkies for this suggestion.) It is then clear that $\Delta_\delta^k g(x-k\delta)$ is decreasing for $1< x< 1+k\delta$ (since $g_k$ is non-increasing). Hence, $f(x)$ is decreasing for $1<x\leq 1 + k \delta$, going from $f(1)=1$ to $f(1+k\delta)=0$. Since $1_{[0,\delta]}^{*k}$ is symmetric around $\delta k/2$ and of integral $\delta^k$, and the value at $x\geq 1$ of the convolution of $g_k$ with $1_{[0,\delta]}^{*k}$ can be written as $k!$ times the integral of $1_{[0,\delta]}^{*k}$ from $0$ to $x-1$, we see that $\frac{\delta^{-k}}{k!}  \Delta_{\delta}^k (1+t) 
= 1- \frac{\delta^{-k}}{k!} \Delta_{\delta}^k (1+k\delta - t)$ for $0\leq t\leq k\delta$, and so $f(1 + t) = 1-f(1+k\delta - t)$ for
$0\leq t\leq k\delta$. It folows that
$$|f-1_{[0,1]}|_1 = \int_1^\infty f(x) dx = \frac{k \delta}{2}.$$
Taking the Mellin transform of $f(x)$ is easy:
$$\begin{aligned}Mf(s) &= \frac{\delta^{-k}}{k!} \sum_{j=0}^{k} (-1)^{k-j} 
\binom{k}{j} M(g(x - (k-j) \delta))(s)  \\ &=
\delta^{-k} \sum_{j=0}^{k} (-1)^{k-j} 
\binom{k}{j} \frac{(1+ (k-j) \delta)^{s+k}}{s (s+1) \dotsc (s+k)}.
\end{aligned}$$
(Here comes the technical difficulty I referred to at first: the derivative of $\frac{(1+ (k-j) \delta)^{s+k}}{s (s+1) \dotsc (s+k)}$ equals $$\frac{(1+ (k-j) \delta)^{s+k}}{s (s+1) \dotsc (s+k)}
\left(\log\left(1 + (k-j)\delta\right) - \frac{1}{s} - \frac{1}{s+1} - \dotsc - \frac{1}{s+k}\right).$$
The term $\log(1+(k-j) \delta)$ is undesirable; in fact, it makes the integral in the statement not converge for $k=1$.
Since, in general, $(Mf)'(s) = M((\log x) f(x))(s)$, it is not surprising that the "logarithmic" variant is more convenient as soon as we have $Mf'$ in the statement.)
Since $t\mapsto t^{k+1}$ is convex-up,
$$\begin{aligned}
\left|\sum_{j=0}^{k} (-1)^{k-j} 
\binom{k}{j} (1+ (k-j) \delta)^{s+k}\right|&\leq
\sum_{j=0}^k \binom{k}{j} (1+(k-j) \delta)^{k+1}\\ &\leq  2^k \left(1 + \frac{k}{2} \delta\right)^{k+1}.
\end{aligned}$$
In other words,
$Mf(s) = \frac{C_{k}(s)}{s (s+1) \dotsc (s+k)}$,
where $$|C_{k}(s)|\leq \left(\frac{2}{\delta}\right)^k 
\left(1 + \frac{k}{2} \delta\right)^{k+1} \sim \left(\frac{2}{\delta}\right)^k.$$
Hence
$$\int_T^\infty |Mf(1+it)|dt \lesssim 
\left(\frac{2}{\delta}\right)^k \int_T^\infty 
\frac{dt}{t^{k+1}} =
\frac{1}{k} \left(\frac{2}{\delta T}\right)^k.$$
Thus, we should choose $\delta$ so as to minimize
$$S_0 = \frac{k \delta}{2} + \frac{c}{k}  \left(\frac{2}{\delta T}\right)^k.$$
Taking derivatives, we see that the minimum is reached for
$$\delta = \frac{2 (c/k)^{\frac{1}{k+1}}}{T^{\frac{k}{k+1}}};$$
then $$S_0 = (k+1) \left(\frac{c T}{k}\right)^{\frac{1}{k+1}} \frac{1}{T}.$$
Now we should choose $k$. Taking derivatives again, we see that we should have
$1 + (k+1) \left(\frac{1}{k+1} \log \frac{c T}{k}\right)'
= 1 - \frac{1}{k+1} \log \frac{c T}{k} - \frac{1}{k}$
close to $0$, i.e.,
$$k\approx \log \frac{c T}{k} + \frac{1}{k}.$$
Taking $k$ to be the integer closest to
$$\log c T - \log \log c T$$
would do. Then we have
$$\begin{aligned}S_0 &\approx (\log c T - \log \log c T + 1) \left(\frac{c T}{\log c T - \log \log c T}\right)^{\frac{1}{\log c T - \log \log c T}} \frac{1}{T} \\ &= \frac{e (\log c T - \log \log c T)
+ O\left(\frac{\log \log c T}{(\log c T)^2}\right)}{T}.\end{aligned}$$
Is this perhaps optimal up to a constant? Or up to its leading term?
(It does seem simple to show it is optimal up to a constant,
and perhaps up to its leading term. To wit: we can consider
a non-symmetric Büthe's "optimal" choice (2014), an integral of the Fourier transform of the function in Logan (1988), rescaled; I put "optimal" within quotation marks because Logan was really minimizing something related but different -- essentially, $\int_T^\infty |Mf(it)| dt$, with the constraint that $f(x)-1_{[0,1]}$ has support on a small interval $[e^{-\epsilon},e^{\epsilon}]$ -- but in the non-symmetric setting, the support does essentially give you $|f(x)-1_{[0,1]}|_1$, so Logan's function has to be essentially optimal in a class of function containing ours. If I say "perhaps" it is because all of this involves so many changes of variables that I still have to get right the constant in front of what one get from Logan!)
