I always found it strange that, in the context of invariant and representation theory, averaging over the group is called the "Reynolds operator". As far as I know the work of Reynolds was in fluid mechanics. He introduced the idea of local averaging in order to distinguish slow and fast variables. As such it is a precursor of things such as the Lyapunov-Schmidt method and Wilson's renormalization group. How did this terminology end up in invariant theory?
I recall this old article in German by Hilbert 1890: Hilbert, David (1890), "Über die Theorie der algebraischen Formen." (http://link.springer.com/article/10.1007%2FBF01208503) where in proof of his theorems on invariant theory he applies the Reynolds operator implying Cayley's omega process. The article is unfortunately in German and I encountered that years ago, irritated as a process engineer who was into fluid mechanics, why Reynolds does into such math stuff. So far I can only answer your question from the other side of the border as engineer, how Hilbert got to Reynolds. I just found also a weak snapshot on this on wiki not that extensive: http://en.wikipedia.org/wiki/Invariant_theory#Hilbert.27s_theorems Nevertheless I think some further reconstruction work would be surely required on illuminating all historic details.
Append moved from my comments: one question however as your title is bit irritating, you are not looking for the authentic name/term Reynolds Operator but the mathematics behind of it, isnt it? Because then one should see back into the history of the term Reynolds Operator and that might be not perfectly synchronous to your time table above or my answer. The name/term Reynolds Operator itself must have been plugged in perhaps later (possibly parallel to Heavyside 1880-87 systematic operational calculus or even later).