With help from Milne's book on étale cohomology, I figured out how answer the question, although I am not sure that this argument is what Artin had in mind, and I still think that there's an easier argument.
There are morphisms of topoi $\pi_X: X_{ET}\rightarrow X_{et}$ from the topos associated
to the big étale site of $X$ to the topos of the small étale site. Similarly for $S$, and
$f:X\rightarrow S$ induces $f^s:X_{et}\rightarrow S_{et}$ and $f^b:X_{ET}\rightarrow
S_{ET}$, and the obvious diagram commutes, i.e. $f^s\pi_X=\pi_Sf^b$.
Given a sheaf $F$ in $S_{ET}$ we get a base change morphism
\[ \pi_S^*R^qf^s_*F\rightarrow R^qf_*^b\pi_X^*F\]
$$ \pi_S^*R^qf^s_*F\rightarrow R^qf_*^b\pi_X^*F$$
Milne calls this "universal base change morphism", for good reasons: Given any morphism $g:S'\rightarrow S$, you also get morphisms of topoi $g^b:S'_{ET}\rightarrow S_{ET}$ and ${g'}^b:X'_{ET}\rightarrow X_{ET}$. Using this to restrict the universal base change morphism to $S'_{ET}$, we get the usual base change morphism for $g$ and $F$. (For this one has to check the commutativity of a few diagrams. All the ingredients can be found, e.g., in great detail in the Stacks Project)
Now, if $F$ is locally constructible, i.e. if the adjunction map $F\rightarrow \pi_X^*\pi_{X,*} F$ is an isomorphism, then it is not hard to check that the universal base change morphism is an isomorphism, and thus every base change morphism is an isorphism.