# The conjecture of Montgomery and Soundararajan on primes in short intervals: Empirical inconsistencies?

Assume that $y/ \log x \rightarrow \infty$ and that $y/x \rightarrow 0$. Then, from a conjecture by Montgomery and Soundararajan, we expect the number of primes in the interval $[x,x+y]$ to be normally distributed with mean $y/\log x$ and standard deviation $\sqrt{y(\log x/y)/(\log x)^2}$. But numerical testing produces a slight and systematic deviation from this conjecture. Specifically, I have tested for different interval lengths $y=x^c$. For each value of $c$, I calculated $\pi(x+y)-\pi(x)$, the number of primes in $[x,x+y]$, for $N$ non-overlapping intervals. Then I normalized the data by subtracting the mean and dividing by the standard deviation corresponding to each interval, as stated by the conjecture. We should therefore expect the resulting $N$ samples to be normally distributed with mean 0 and standard deviation $\sigma=1$. However, what I get is the following:

\begin{matrix} c& \sigma& N\\ \hline 0.20 & 0.967 & 240000\\ 0.25 & 0.966 & 240000\\ 0.30 & 0.965 & 240000\\ 0.40 & 0.958 & 240000\\ 0.50 & 0.947 & 240000\\ 0.55 & 0.917 & 40000\\ 0.60 & 0.899 & 20000\\ 0.65 & 0.891 & 10000\\ 0.70 & 0.889 & 5000\\ \end{matrix}

While each data set indeed are normally distributed, what appears to be happening is that the standard deviation decreases with increasing $c$, as compared to the conjecture by Montgomery and Soundararajan. The numerical results seem to be reasonably consistent, so I don't think they are an artifact of sampling only a finite number of intervals (however, please point out if I did any apparent mistakes in the above). I am not fluent enough in the theory to walk through Montgomery and Soundararajan's arguments myself, so I would greatly appreciate any comments on or explanation of this finding.

EDIT Including the lower order term in Montgomery and Soundararajan's conjecture, as suggested in the answer below by Lucia, we have the following revised numerics:

\begin{matrix} c& \sigma& N\\ \hline 0.20 & 1.000 & 240000\\ 0.25 & 1.001 & 240000\\ 0.30 & 1.003 & 240000\\ 0.40 & 1.002 & 240000\\ 0.50 & 1.000 & 240000\\ 0.55 & 1.001 & 40000\\ 0.60 & 0.992 & 20000\\ 0.65 & 0.995 & 10000\\ 0.70 & 1.009 & 5000\\ \end{matrix}

These numbers strongly support Montgomery and Soundararajan's conjecture, so it is clear that the missing lower order term was indeed responsible for the observed discrepancy.

There are lower order terms in the work of Montgomery and Soundararajan that may account for the discrepancies you're observing. If you look at Theorem 3 of the paper that you linked, you'll find that the standard deviation should really be $$\frac{\sqrt{y (\log \frac xy +B)}}{\log x},$$ where $B=1-\gamma-\log (2\pi)=-1.415\ldots$. This is asymptotically the same as what you have, but numerically the second order term can make a difference. Note also that the lower order term becomes more significant as $y$ gets larger, which is a feature that you see in your data.
So this is really a question for you: whether taking the new standard deviation with lower order terms gives you values for $\sigma$ closer to $1$. I'd be curious to know the revised numerics.