I recall this old article in German by Hilbert 1890: Hilbert, David (1890), "Über die Theorie der algebraischen Formen." (http://link.springer.com/article/10.1007%2FBF01208503) where in proof of his theorems on invariant theory he applies the Reynolds operator implying Cayley's omega process. The article is unfortunately in German and I encountered that years ago, irritated as a process engineer who was into fluid mechanics, why Reynolds does into such math stuff. So far I can only answer your question from the other side of the border as engineer, how Hilbert got to Reynolds. I just found also a weak snapshot on this on wiki not that extensive: http://en.wikipedia.org/wiki/Invariant_theory#Hilbert.27s_theorems Nevertheless I think some further reconstruction work would be surely required on illuminating all historic details.
Append moved from my comments: one question however as your title is bit irritating, you are not looking for the authentic name/term Reynolds Operator but the mathematics behind of it, isnt it? Because then one should see back into the history of the term Reynolds Operator and that might be not perfectly synchronous to your time table above or my answer. The name/term Reynolds Operator itself must have been plugged in perhaps later (possibly parallel to Heavyside 1880-87 systematic operational calculus or even later).