I've always been fascinated by the fact that the classical Gauss sum has absolute value $\sqrt p$, which is exactly what we would expect if we were to interpret the Gauss sum as a random walk. In particular, I have long wondered whether this apparent link between number theory and probability theory is just a curiosity, or whether it is a symptom of a deeper connection. For example, there are many heuristic arguments in number theory that treat primes as being "random." Is there any example of such a heuristic argument that can be made rigorous by using the above observation about Gauss sums?

The thing is that it is well known that for the quadratic Gauss sums, expressed as an exponential sum rather than with Legendre symbols, the path is very much not a random walk when you plot it in the complex plane. There is plenty of structure visible as approximate Cornu spirals. Such sums are not the only Gauss sums, as I know to my cost; and the quadratic case is atypical (perhaps). But from a highflown point of view, a Gauss sum is a special function in the theory of finite fields (like a Gamma function). There seems to be more mileage in asking about what is special about it. 


Caveat lector: as an answer to Timothy's question, this is tangential at best. Regarding Gerry and Kevin's comments, people might be interested in Section 2 of these notes of mine from an undergraduate number theory course, in which the philosophy of "almost square root error" is batted around for a while, especially with regard to the Riemann hypothesis. This is maybe my favorite set of lecture notes from this course, perhaps because I got a chance to (talk and) write excitedly about things I hardly understand: I certainly do not claim a professional level of insight here. (Indeed some regulars on this site could do far better, and I would be interested to hear their critiques.) The upshot though is that of agreement with Gerry and Kevin: having size $\sqrt{p}$ on the nose is just a little bit better than random, but the little bit makes a big difference: to me this suggests very precise structure rather than randomness. Please see here for another take on the subject of almost square root error in the context of the SatoTate Conjecture. This latter article was written at almost the same time as mine, and the author happens to be my former thesis advisor. I am reasonably sure that both of these coincidences are indeed coincidental. 


Actually I would think there are connections. Even if the the coincidence $\sqrt{p}$ seems ordinary, there are lowcorrelation sequences which owe their low correlation to Gauss sum estimates and looking from a probabilistic view point sequences, I would think there would be interpretations from a view point of uncorrelated random variables. Your starting point could be lowcorrelation sequences used in communications systems and beyond that I would think Nicholas Katz and Sarnak's dive into random matrices would help. http://books.google.com/books?id=wXyOPbzvowsC&printsec=frontcover&dq=inauthor:%22Nicholas+M.+Katz%22&hl=en&ei=j9IrTu24CpKnsAKG5c3DCw&sa=X&oi=book_result&ct=result&resnum=5&ved=0CD8Q6AEwBA#v=onepage&q&f=false 

