Illusie's article about étale cohomology available here (in French) mentions that the standard definition of compactly supported cohomology (and higher direct images with compact support) does not give the right answers in the case of étale cohomology.

He says Grothendieck had the idea of defining them as follows instead: $$Rf_! = Rg_* \circ j_!$$ where $f = gj$ is a compactification of $f$: $g$ is proper, and $j$ is an open immersion. It takes a bit of work to see that this is well defined (a theorem of Nagata guarantees the existence of this compactification, and the proper base change theorem shows that the result does not depend on which compactification is chosen).

What is the explanation here for the difference between Grothendieck's definition and the usual one, which amounts to $Rf_! = R(g_* j_!)$ ? What is it that allows us to say that either is the "right" one?

It seems to me that a satisfactory explanation should be more than just computing the groups in a special case and appealing to intuition (indeed, the compactly supported cohomology groups are often strange, at first sight, even in the topological situation).

For example, I could imagine that an explanation might involve derived categories: after all, one of the main uses of $Rf_!$ is Verdier duality, which implies many useful properties such as Poincaré duality. This maybe gives some hints as to which definition is preferred.