I recall this old article in German by Hilbert 1890: Hilbert, David (1890), "Über die Theorie der algebraischen Formen." (http://link.springer.com/article/10.1007%2FBF01208503) where in proof of his theorems on invariant theory he applies the Reynolds operator implying Cayley's omega process. The article is unfortunately in German and I encountered that years ago, irritated as a process engineer who was into fluid mechanics, why Reynolds does into such math stuff. So far I can only answer your question from the other side of the border as engineer, how Hilbert got to Reynolds. I just found also a weak snapshot on this on wiki not that extensive: http://en.wikipedia.org/wiki/Invariant_theory#Hilbert.27s_theorems
Nevertheless I think some further reconstruction work would be surely required on illuminating all historic details.

*Append moved from my comments:* one question however as your title is bit irritating, you are not looking for the authentic name/term *Reynolds Operator* but the mathematics behind of it, isnt it? Because then one should see back into the history of the term *Reynolds Operator* and that might be not perfectly synchronous to your time table above or my answer. The name/term *Reynolds Operator* itself must have been plugged in perhaps later (possibly parallel to Heavyside 1880-87 *systematic operational calculus* or even later).