In the special case when $r=1$, Montgomery and Odlyzko [3] observed that this optimization problem had already been solved using prolate spheroidal wave functions (see comments below). Here is how they reduced the problem to one that was already solved. In this case, we have $$ M(c) = c+ \int_0^1\int_0^1 f(u) f(v) \frac{\sin(\pi c(u-v))}{\pi(u-v)} \ dv \ du. $$ The double integral on the right-hand side is $$ \int_{-c/2}^{c/2} \left| \int_0^1 f(v) e^{2\pi i v} dv \right|^2 dt :=I(c),$$ say. They then observed that choosing $$ f(x) = aR_{00}^{(1)}[\pi c/2,2x-1]$$ maximizes $I(c)$, where $R_{mn}^{(1)}[c,x]$ is the radial prolate spheroidal wave function of the first kind of order m and degree n, and a is a constant to be chosen according to our above normalization.

Theorem: Let $\alpha(c)$ be the least number such that $$ \int_{-c/2}^{c/2} \left|\int_0^1 f(x) e^{2\pi i x} dx \right|^2 dt \le \alpha(c) \int_0^1 |f(x)|^2 dx $$ for all $f\in L^2[0,1]$. Then $\alpha(c)$ is strictly increasing, $\alpha(c)\lt c$ for $c \gt 0$, $\alpha(c)\sim c$ as $c\to 0^+$, and $\alpha(c)\to 1$ as $c\to \infty$. Moreover, equality is achieved in the above inequality if $f(x) = R_{00}^{(1)}[\pi c/2,2x-1]$.

In the special case when $r=1$, Montgomery and Odlyzko [3] observed that this optimization problem had already been solved using prolate spheroidal wave functions (see comments below). Here is how they reduced the problem to one that was already solved. In this case, we have $$ M(c) = c+ \int_0^1\int_0^1 f(u) f(v) \frac{\sin(\pi c(u-v))}{\pi(u-v)} \ dv \ du. $$ The double integral on the right-hand side is $$ \int_{-c/2}^{c/2} \left| \int_0^1 f(v) e^{2\pi i t v} dv \right|^2 dt :=I(c),$$ say. They then observed that choosing $$ f(x) = aR_{00}^{(1)}[\pi c/2,2x-1]$$ maximizes $I(c)$, where $R_{mn}^{(1)}[c,x]$ is the radial prolate spheroidal wave function of the first kind of order m and degree n, and a is a constant to be chosen according to our above normalization.

Theorem: Let $\alpha(c)$ be the least number such that $$ \int_{-c/2}^{c/2} \left|\int_0^1 f(x) e^{2\pi i t x} dx \right|^2 dt \le \alpha(c) \int_0^1 |f(x)|^2 dx $$ for all $f\in L^2[0,1]$. Then $\alpha(c)$ is strictly increasing, $\alpha(c)\lt c$ for $c \gt 0$, $\alpha(c)\sim c$ as $c\to 0^+$, and $\alpha(c)\to 1$ as $c\to \infty$. Moreover, equality is achieved in the above inequality if $f(x) = R_{00}^{(1)}[\pi c/2,2x-1]$.

In the special case when $r=1$, Montgomery and Odlyzko [3] actually solved the optimization problem using prolate spheroidal wave functions. Here is the basic idea of their argument. In this case, we have $$ M(c) = c+ \int_0^1\int_0^1 f(u) f(v) \frac{\sin(\pi c(u-v))}{\pi(u-v)} \ dv \ du. $$ The double integral on the right-hand side is $$ \int_{-c/2}^{c/2} \left| \int_0^1 f(v) e^{2\pi i v} dv \right|^2 dt :=I(c),$$ say. They then observed that choosing $$ f(x) = aR_{00}^{(1)}[\pi c/2,2x-1]$$ maximizes $I(c)$, where $R_{mn}^{(1)}[c,x]$ is the radial prolate spheroidal wave function of the first kind of order m and degree n, and a is a constant to be chosen according to our above normalization.

[3] H. L. Montgomery and A. M. Odlyzko, Gaps between zeros of the zeta function, Colloq. Math. Soc. Janos Bolyai, 34. Topics in Classical Number Theory (Budapest, 1981), North-Holland, Amsterdam, 1984.

In the special case when $r=1$, Montgomery and Odlyzko [3] observed that this optimization problem had already been solved using prolate spheroidal wave functions (see comments below). Here is how they reduced the problem to one that was already solved. In this case, we have $$ M(c) = c+ \int_0^1\int_0^1 f(u) f(v) \frac{\sin(\pi c(u-v))}{\pi(u-v)} \ dv \ du. $$ The double integral on the right-hand side is $$ \int_{-c/2}^{c/2} \left| \int_0^1 f(v) e^{2\pi i v} dv \right|^2 dt :=I(c),$$ say. They then observed that choosing $$ f(x) = aR_{00}^{(1)}[\pi c/2,2x-1]$$ maximizes $I(c)$, where $R_{mn}^{(1)}[c,x]$ is the radial prolate spheroidal wave function of the first kind of order m and degree n, and a is a constant to be chosen according to our above normalization.

[3] H. L. Montgomery and A. M. Odlyzko, Gaps between zeros of the zeta function, Colloq. Math. Soc. Janos Bolyai, 34. Topics in Classical Number Theory (Budapest, 1981), North-Holland, Amsterdam, 1984.

Edit/Additional Comments: The solution optimization problem when $r=1$ should probably be attributed to Slepian and Pollak and to Landau and Pollack in Prolate spheroidal wave functions, Fourier analysis and uncertainty I and II, Bell System Tech. J. (1961) 40, pp. 43-61 (I) and pp. 65-84 (II). Among other things, they prove the following results.

Theorem: Let $\alpha(c)$ be the least number such that $$ \int_{-c/2}^{c/2} \left|\int_0^1 f(x) e^{2\pi i x} dx \right|^2 dt \le \alpha(c) \int_0^1 |f(x)|^2 dx $$ for all $f\in L^2[0,1]$. Then $\alpha(c)$ is strictly increasing, $\alpha(c)\lt c$ for $c \gt 0$, $\alpha(c)\sim c$ as $c\to 0^+$, and $\alpha(c)\to 1$ as $c\to \infty$. Moreover, equality is achieved in the above inequality if $f(x) = R_{00}^{(1)}[\pi c/2,2x-1]$.

One of the possible hang-ups in solving the optimization problem when $r>1$ is that I have not been able to "complete" the double integral $$\int_0^1 (1-u)^{r^2-1}f(u) \int_0^u \frac{\sin(\pi c v)}{\pi v} f(u-v) \ dv \ du$$ into a double integral of the form $$ \int_0^1 \int_0^1 [\text{nice integrand}] \ dv \ du,$$ which is the first step in Montgomery & Odlyzko's argument.