What heuristic arguments support Montgomery's conjecture for primes in short intervals?

Question

I have a question regarding a conjecture due to H. L. Montgomery on the number of primes in short intervals. The conjecture apparently arises from probabilistic reasoning upon assuming the Riemann Hypothesis and some statistical randomness across the ordinates $\gamma$ of the nontrivial zeros of the Riemann zeta function. Let $\psi(x)=\sum_{n\le x}\Lambda(n)$ denote the (second) Chebyshev function. Then (considering only $\varepsilon>0$), he makes the following

Conjecture. (Montgomery) $$\psi(x+h)-\psi(x)=h+O_{\varepsilon}(h^{\frac{1}{2}}x^{\varepsilon})$$ for $2\le h\le x$.

My question: Which heuristic arguments support this conjecture (excluding numerical verification)?

The conjecture appears in several places:

(A) H. L. Montgomery, "Problems concerning prime numbers", Proceedings of symposia in pure mathematics, Vol. XXVIII, pp.307-310, Mathematical developments arising from Hilbert Problems (1976), AMS. Providence, Rhode island. [See p.309]

(B) D. A. Goldston, "On a result of Littlewood concerning prime numbers", Acta Arithmetica, Vol.40 (1982), pp. 263-271. [See p.269]

(C) H. L. Montgomery, R. C. Vaughan, Multiplicative Number Theory I. Classical Theory, Cambridge University Press (2007) [Conjecture 13.4, p.422]

Following the argument of Montgomery and Vaughan (p.422), is something I manage up to a certain point, but I'm not sure how to "get there". Specifically, using the explicit formula for $\psi(x)$ on the form $$\psi(x)=x-\sum_{\rho}\frac{x^{\rho}}{\rho}-\frac{\zeta'}{\zeta}(0)-\frac{1}{2}\log(1-x^{-2})+\frac{1}{2}\Lambda(x) \hspace{5mm}(x>1),$$ we can write $$\psi(x+h)-\psi(x)=h-\sum_{|\gamma|\le T}C(\rho)+\lim_{U\to \infty}\sum_{T<|\gamma|\le U}C(\rho)+ O\big(\hspace{-0.2mm}\log\hspace{0.4mm}\max(x,h)\big)$$ for $x,h\ge 2$, say, where $$C(\rho)=\frac{(x+h)^{\rho}-x^{\rho}}{\rho} \ll \min\Big(hx^{\beta-1}, \frac{x^{\beta}}{|\gamma|}\Big).$$

Now assume that the Riemann hypothesis is true, and write \begin{align*}C(\rho)=&\;\int_{x}^{x+h}t^{\rho-1}dt=\int_{x}^{x+h}x^{\rho-1}dt+\int_{x}^{x+h}t^{\rho-1}-x^{\rho-1}dt\\ =&\; hx^{\rho-1}+\int_{0}^{h}(x+t)^{\rho-1}-x^{\rho-1}dt\\ =&\; hx^{\rho-1}+\int_{0}^{h}(\rho-1)\int_{0}^{t}(x+z)^{\rho-2}dzdt\\[1mm] =&\; hx^{-\frac{1}{2}+\gamma i}+O\big(h^2x^{-\frac{3}{2}}|\gamma|\big). \hspace{30mm} (\dagger) \end{align*} Ignoring the error here for the moment and taking $T=x/h$, then $$\sum_{|\gamma|\le x/h}hx^{-\frac{1}{2}+\gamma i}=\frac{h}{\sqrt{x}}\sum_{|\gamma|\le x/h}\text{e}^{\gamma \log(x)i}.$$ Now, if we were to replace the $\gamma \log x$-s with independent and identically distributed uniform random variables $(Y_n)_{n=1}^{\infty}$ on the interval $[0,2\pi)$ , then, as this this post shows, we could conjecture that the above sum behaves like $$\frac{h}{\sqrt{x}}\sum_{n\ll N(x/H)}\text{e}^{Y_n i}\ll_{\varepsilon} \frac{h}{\sqrt{x}}N\big(\frac{x}{h}\big)^{1/2+\varepsilon}\ll_{\varepsilon} \frac{h}{\sqrt{x}}\Big(\frac{x}{h}\log\big(\frac{x}{h}\big)\Big)^{\frac{1}{2}+\varepsilon}=h^{\frac{1}{2}-\varepsilon}x^{\varepsilon}\big(\hspace{-0.2mm}\log \frac{x}{h}\big)^{\frac{1}{2}+\varepsilon},$$ as in Montgomery's conjecture.

This is the point where I get stuck. For indeed, there are two unresolved sizes here. The first is the contribution of the large zeros: $$\sum_{|\gamma|>x/h}C(\rho), \hspace{15mm} (\ddagger)$$ and the second is the contribution of the error term in $(\dagger)$. The contribution of the error in $(\dagger)$ may not be too hard to resolve, but I am quite unable to show that the sum $(\ddagger)$ is of order $O_{\varepsilon}(h^{\frac{1}{2}}x^{\varepsilon})$. For example, using the explicit formula for $\psi(x)$ on the (commonly stated) form $$\psi(x)=x-\sum_{\rho}\frac{x^{\rho}}{\rho}+O\Big(\frac{x\log^2(xT)}{T}+\log x\Big) \hspace{4mm} (x,T\ge 2),$$ I obtain $$\sum_{|\gamma|>x/h}C(\rho)\ll \frac{(x+h)\log^2((x+h)\frac{x}{h})}{x/h}\ll h\log^2 x$$ if $2\le h\le x$, which is not $O_{\varepsilon}(h^{\frac{1}{x}}x^{\varepsilon})$ unless $h\ll x^{2\varepsilon}(\log x)^{-4}$ (and this does not permit $2\le h\le x$ if, say, $0<\varepsilon<1/2$). The problem may be that the explicit formula with error term used here is unconditional, and that a better formula assuming RH should be employed.

Montgomery and Vaughan say about this (on p.422), that "The contribution of zeros with $|\gamma|>x/h$ can be attenuated by employing a smoother weight, but no amount of smoothing will eliminate the smaller zeros." By smoothing, they here likely mean that the explicit formula for $\sum_{n\le x}\Lambda(n)=\sum_{n=1}^{\infty}\Lambda(n)1_{(x,x+h]}(n)$ should be replaced by an explicit formula for $$\sum_{n=1}^{\infty}\Lambda(n)w(n),$$ where $w(n)=w(n;x,h)$ is a `weight function'. This weight function should be such that it gives a useful explicit formula (meaning that the contribution of the $\rho$-s in the right hand side decays rapidly as a function of $|\gamma|$), but also approximates the indicator function $1_{(x,x+h]}(n)$ to such an extent that $\sum_{n}\Lambda(n)w(n)$ is close to $\psi(x+h)-\psi(x)$.

I have been playing around with such explicit formulas lately, including the formulas \begin{align*} \frac{1}{k!}\sum_{n\le x}\Lambda(n)(x-n)^{k}=&\; \frac{x^{k+1}}{(k+1)!}-\frac{x^{k}}{k!}\frac{\zeta'}{\zeta}(0)-\sum_{\rho}\frac{x^{\rho+k}}{\rho(\rho+1)\cdots (\rho+k)}+ \sum_{0\le j\le (k-1)/2}\frac{x^{k-2j-1}}{(2j+1)!(k-2j-1)!}\frac{\zeta'}{\zeta}(-2j-1)\\[1.5mm] +&\; (-1)^{k}\sum_{j>k/2}x^{k-2j}\frac{(2j-k-1)!}{(2j)!}+\sum_{0<j\le k/2} \frac{x^{k-2j}}{(2j)!(k-2j)!}\Big(\frac{1}{2}\frac{\zeta''}{\zeta'}(-2j)-\log x+\sum_{\substack{r=-2j\\ r\ne 0}}^{k-2j}r^{-1}\Big) \hspace{5mm} (x\ge 1, k\in \mathbb{N}^{+}), \\[2mm] \frac{1}{\Gamma(\xi+1)}\sum_{n<x}\Lambda(n)(x-n)^{\xi}=&\; \frac{x^{\xi+1}}{\Gamma(\xi+2)}-\sum_{\rho}\frac{x^{\rho+\xi}\Gamma(\rho)}{\Gamma(\rho+\xi+1)}-\frac{x^{\xi}}{\Gamma(\xi+1)}\frac{\zeta'}{\zeta}(0)+\sum_{j=0}^{\infty} \frac{x^{\xi-2j-1}}{\Gamma(2j+2)\Gamma(\xi-2j+1)}\cdot \frac{\zeta'}{\zeta}(-2j-1)\\ -&\;\sum_{j=1}^{\infty} \frac{x^{\xi-2j}}{\Gamma(2j+1)\Gamma(\xi-2j+1)}\Big(-C_{\text{Eul}}+\sum_{k=1}^{2j}\frac{1}{k}+\frac{1}{2}\frac{\zeta''}{\zeta'}(-2j)+\log(x)-\psi^{(0)}(\xi-2j+1)\Big) \hspace{5mm} (x\ge 1, \text{Re }\xi>0, \xi \not \in \mathbb{Z}), \\[2mm] \sum_{n\le x}\Lambda(n)\log(x/n)=&\;x-\sum_{\rho}\frac{x^{\rho}}{\rho}-(\log 2\pi)\log x-(\frac{\zeta'}{\zeta})'(0)-\frac{1}{4}\sum_{k=1}^{\infty}\frac{x^{-2k}}{k^2} \hspace{5mm} (x>1),\\[2mm] \sum_{n=1}^{\infty}\Lambda(n)\text{e}^{-n/z}=&\; z-\sum_{\rho}\Gamma(\rho)z^{\rho}-\text{e}^{-1/z}\log 2\pi -(-1+\cosh 1/z)\log z+\sum_{k=1}^{\infty}(-1)^{k}\frac{\zeta'}{\zeta}(k+1)\frac{z^{-k}}{k!}\\ -&\;\sum_{k=0}^{\infty}\frac{\Gamma'}{\Gamma}(2k+2)\frac{z^{-2k-1}}{(2k+1)!} \hspace{5mm} (\text{Re }z>0). \end{align*} However, I am not able to get the desired result. Indeed, I am not able to combine the weight functions provided in the formulas above, to get a weight function approximating $1_{(x,x+h]}(n)$ to a reasonable extent while giving a reasonable explicit formula. This may be because:

1. I have not found any literature explaining what would constitute a good smoothing of $1_{(x,x+h]}(n)$ in this case (i.e. how does the smoothness and cut-off play a role)
2. Taking a weighted sum of one of the explicit formulas above, one could likely approximate $1_{(x,x+h]}(n)$ by expressions of the form $\sum_{n=1}^{N}a_nw(n,x_n,h_n)$. Here I am not at all sure which function spaces I would try to do an approximation in (e.g. which norm).

These investigations are related to my Master's thesis on primes in short intervals, and I would highly appreciate if anyone could comment on how Montgomery's conjecture can be backed up. (And possibly also what the goal/strategy of the smoothing process should be, if you would be kind enough to explain).

Sincerely, R.

Answer 1

The motivation for many conjectures about the distribution of primes $p\in[x,x+h]$ in intervals of length $o(\sqrt{x}\log x)$ comes from studying the mean square of $|\psi(x+h)-\psi(x)-h|$. For example, for $\theta\in[0,1]$, Selberg considered the integral

$$J(x,\theta):= \frac{1}{x}\int_{x}^{2x}|\psi(t+\theta t)-\psi(t)-\theta t|^2 dt.$$

Selberg proved that RH suffices to conclude that if $1/x\leq\theta\leq x$, then

$$J(x,\theta) \ll x\theta (\log (2/\theta))^2,$$

which means that if $x$ is large and $1/x\leq \theta\leq x$, then for all $t\in[x,2x]$ outside of a subset of Lebesgue measure $o(x)$, we have

$$\psi(t+\theta t)-\psi(t)=\theta t +O(\sqrt{\theta x}\log x).$$

One might now conjecture that the subset of Lebesgue measure $o(x)$ is in fact empty. However, this contradicts a result of Maier, who used the Maier matrix method to prove that if $A>1$, then

$$\limsup_{x\to\infty}\frac{\psi(x+(\log x)^A)-\psi(x)}{(\log x)^{A}}>1$$

and

$$\liminf_{x\to\infty}\frac{\psi(x+(\log x)^A)-\psi(x)}{(\log x)^{A}}<1.$$

So what is the correct minimal length in which we obtain an asymptotic prime number theorem? Is it $\exp((\log x)^{1/3})$? $\exp((\log x)^{1/2})$? $\exp((\log x)^{3/4})$? There is no "scientific" way to determining what the right threshold is (at least that I am aware of), so at this point we conjecture that whatever the correct threshold should be (that is, the least permissible value of $\theta x$), it should still be less than $x^{\epsilon}$ for any fixed $\epsilon>0$. This is an alternate formulation of Montgomery's conjecture in the original post.

Answer 2

It is worth noting that Montgomery's conjecture can be derived from standard probabilistic models of the primes without needing to say anything about the zeta zeros.

The simplest such model is called the Cramér random model. The setup is as follows: consider a sequence of independent random variables $(Y_n)_{n\in\mathbb{N}}$ such that $$\mathbb{P}(Y_n=\log n)=\frac{1}{\log n}, \quad \mathbb{P}(Y_n=0)=1-\frac{1}{\log n}.$$ Then the random sum $\sum_{x < n\leq x+h} Y_n$ is a model for $\psi(x+h)-\psi(x)$. Note that $\mathbb{E}\sum_{x < n\leq x+h} Y_n = h$ for any integers $2\leq h \leq x$. We'd like to show that this random sum is within $O_\varepsilon(h^{1/2}x^\varepsilon)$ of its mean almost surely. (In other words, Montgomery's conjecture holds in the Cramér random model almost surely.)

There are various methods for showing that a random sum is typically close to its mean. In this case, we can use (for example) Hoeffding's inequality to get $$\mathbb{P}(|\sum_{x < n\leq x+h} Y_n - h| \geq \lambda h^{1/2}) \leq 2\exp(-\lambda^2/(\log x)^2)$$ for any $\lambda>0$. Note that if we take $\lambda=x^\varepsilon$, then this bound decays quite rapidly in $x$, and indeed it is summable if we sum over all pairs of integers $h,x$ with $2 \leq h \leq x$. Consequently, by the Borel-Cantelli lemma, we have $$|\sum_{x < n\leq x+h} Y_n - h| < \lambda h^{1/2}= x^\varepsilon h^{1/2}$$ for all but finitely many choices of $h$ and $x$ almost surely. In other words, Montgomery's conjecture holds almost surely.

Limitations of the Cramér model: Note that the choice of $\lambda = x^\varepsilon$ in above argument was more than sufficient to be able to apply Borel-Cantelli. Indeed, even $\lambda = (\log x)^{3/2+\varepsilon}$ would work. So should Montgomery have conjectured that $\psi(x+h)-\psi(x)=h + O_\varepsilon((\log x)^{3/2+\varepsilon} h^{1/2})?$ The answer is no because this is known to be false by Maier's theorem. This is a well-known limitation of the Cramér random model. To make sense of what's actually going on in very short intervals one needs to account for divisibility by small primes in the random model.