I read somewhere that Hilbert wanted to prove Riemann hypothesis using Fredholm's work on integral equations but I can't find anything online.
Perhaps the best documentation of that story, together with Pólya's story of his similar response to a question from Landau, is at Andrew Odlyzko's page.
In those days, prior to Stone's and vonNeumann's work on unbounded operators, their resolvents were much more tractable, proof-wise. So, Hilbert's, Schmidt's, Fredholm's, Volterra's, and others' work on integral operators (often compact) were unassailably rigorous, in contrast to discussion of differential operators and other unbounded operators.
The general idea of finding/making self-adjoint operators whose eigenvalues are $s(s-1)$ for zeros $s$ of $\zeta$ (or similar…), has two general approaches. One is to make operators whose eigenvalues are parametrized in that way, and then show that they're self-adjoint. Alain Connes is a notable proponent of this approach.
The opposite approach (in which I've had some interest for a while) is to make "natural" self-adjoint operators (descended from Laplace–Beltrami operators on automorphic geometric objects) whose discrete spectrum… if any … is related to zeros of zeta or other L-functions. This was pioneered in papers of Y. Colin de Verdiere in 1981-83. Since 2011, E. Bombieri and I have/had worked to be sure that various ideas supporting this could be made rigorous… and, sadly, showing that Montgomery's pair-correlation (and/or similar) showed that (unless RH is false!?!) not all the zeros of zeta can participate in the discrete spectrum of these operators (Ref. Bombieri and Garrett "Designed Pseudo-Laplacians", specifically, no more than $94\%$.)
Since this question was bumped to the front page, I might address
Q1: Can someone provide historical references for it?
This goes back to André Weil, who writes in [1] that Ernst Hellinger, a student of Hilbert, told him how Hilbert proposed at his seminar in Göttingen that Fredholm’s results on integral equations could lead to a solution of the Riemann hypothesis:
After having demonstrated that the eigenvalues of a symmetric kernel are real, Hilbert is quoted as saying: “Et avec ce théorème, Messieurs, nous démontrerons l’hypothèse de Riemann”. (Weil writes in French, the original must have been in German.)
[1] A. Weil. Commentaire ‘Sur les “formules explicites” de la théorie des nombres premiers’. In: Scientific works, volume II, page 527.
