If we have a map $f : \mathbb P^1_R \to \mathbb P^1_R$ over $\operatorname{Spec}(R)$, with $R$ a commutative ring, which we assume to be etale, then is it possible to characterize $f$? Must it be an automorphism? Does it help to assume $R$ is a field or $\mathbb Z$ or …?
I think the analogous question for $\mathbb A^1_R$ would be no because of positive characteristic phenomena.
This is true. First note that $f$ is proper [Tag 01W6 (2)] and locally quasi-finite [Tags 03WS], hence finite [Tag 02LS]. Since $f$ is flat [Tag 02GS] and finitely presented, it is finite locally free [Tag 02KB]. It suffices to show that $f$ is locally free of rank $1$ by an easy argument (see here for example).
Now if $S$ is a scheme and $\mathscr F$ is a finite locally free $\mathcal O_S$-module, in order to show that $\mathscr F$ has rank 1, it suffices to show that the same holds for $\mathscr F \otimes \kappa(s)$ for all $s \in S$. Thus, we reduce to the case where $R$ is a field, where the result is an easy consequence of the Riemann–Hurwitz formula (alternatively, use some étale Galois theory and a connectedness theorem; see e.g. here).
Remark. The analogous result for $\mathbf A^1_R$ is false, even if you assume a priori that $f$ is finite. Indeed, for $R = \mathbf F_p$ there is an Artin–Schreier cover $\mathbf A^1 \to \mathbf A^1$ given by $x \mapsto x^p-x$.
However, the result for $\mathbf A^1_R$ is still true as long as each connected component of $R$ has a point with residue characteristic $0$ (for example if $R$ is a domain of characteristic $0$, e.g. $R = \mathbf Z$). Indeed, since the rank of a locally free sheaf is locally constant, it suffices to run the argument above for one point in each component.
I believe that when $R = k$ is a field $f$ must be an isomorphism. Indeed, because $\mathbb{P}^1_k$ is a smooth curve, we can apply the Riemann-Hurwitz formula to write $$-2 = -2\deg(f)+\sum_{y \in \mathbb{P}^1_k}(k_y-1) $$ where $k_y$ is the ramification index of $f$ at $y$. Because $f$ is étale, it is unramified, so $k_y=1$ and the sum vanishes. We now read off $\deg(f)=1$ and $f$ is an isomorphism, as claimed.
