Formally étale at all primes does not imply formally étale? All rings are assumed to be commutative and unital, with all homomorphisms unital as well.
On last week's homework, there was a mistake in one of the questions:


(2.5) Let $R\to S$ be a morphism of commutative rings giving $S$ an $R$-algebra structure. Suppose that the induced maps $R\to S_{\mathfrak{p}}$ are formally étale for all prime ideals $\mathfrak{p}\subset S$. Then $R\to S$ is formally étale.  


According to our professor, the exercise should have stated additionally that $S$ was finitely presented over $R$ (which allows us to prove that $S$ is in fact étale over $R$ rather than just formally étale).  This is not too hard to do and is left as an exercise.  (You can also find it in EGA).  
However, I'm interested in seeing either a counterexample or a proof for the stronger claim in the grey box.  
 A: Using the module of Kähler differentials, it is easy to show that $R\to S$ is formally unramified if and only if the induced maps $R\to S_{\mathfrak{p}}$ are formally unramified for all primes $\mathfrak{p}\subset S$.
Consider a presentation of $S$ over $R$ as $R[X]/I$ in generators and relations, where $R[X]:=R[X_m]_{m\in M}$ is a polynomial ring in a possibly infinite family of indeterminates indexed by $M$, and $I\subset R[X]$ is an ideal.  Fix a family of generators of $I=(F_j)_{j\in J}$ indexed by $J$, again not necessarily finite.
It is enough to show that $R\to S$ is formally smooth.  This is equivalent to showing that there exists a morphism of $R$-algebras that is a splitting for the canonical projection $\pi:R[X]/I^2 \to R[X]/I=S$, which will necessarily be unique because $R\to S$ is formally unramified. 
Let $\overline{X}_m$ denote the image of $X_m$ in $R[X]/I^2$.  We must find elements $\delta_m\in I/I^2$ such that $(\forall j\in J)F_j(X_m + \delta_m)=0$.  We rewrite this using Taylor's formula as $$\bar{F}_j+ \sum_{m\in M}\overline{\frac{\partial F_j}{\partial X_m}}\delta_m=0.$$
Rearranging, we get a system of equations indexed by $J$
$$(*)_{j\in J} \qquad \sum_{m\in M}\overline{\frac{\partial F_j}{\partial X_m}}\delta_m=-\overline{F}_j.$$
We wish to find a unique solution for this system in the $\delta_m$.  Since $\Omega_{S/R}=0$, each $dX_m\in \Omega_{R[X]/R}$ is an $S$-linear combination $dX_m=s_{m,1}dF_{j_{m,1}}+\cdots + s_{m,h_m}dF_{j_{m,h_m}}$.  If we use the $s_{m,k}$ as coefficients to form $S$-linear combinations of the equations $(*)_{j_k}$, for each $m$, we get an equation of the form $$(**)_m \qquad \delta_m=-(s_{m,1}\overline{F}_{j_{m,1}}+\cdots + s_{m,h_m}\overline{F}_{j_{m,h_m}}).$$
Showing that these define solutions for all of the equations $(*)_j$ is not immediate, but it is a local question on $S$.  However, our local rings $S_{\mathfrak{p}}$ are all formally étale, so the local conditions are satisfied.  Then this proves the global claim.
(Note: This is not my proof.  I've paraphrased the proof communicated to me by Mel Hochster.)
Edit: Fixed LaTeX using Scott's suggestion.
A: EDIT: Don't bother reading my partial solution.  Brian Conrad pointed out that an easier way to do what I did is to use the equivalent definition of formally unramified in terms of Kähler differentials.  And later on, fpqc posted below a complete solution passed on by Mel Hochster, who got it from Luc Illusie, who got it from ???.
OLD ANSWER:
Here is a half-answer.  I'll prove half the conclusion, but on the plus side I'll use only half the hypothesis!  Namely, I will prove that if $R \to S_{\mathfrak{p}}$ is formally unramified for all primes $\mathfrak{p} \subset S$, then $R \to S$ is formally unramified.
Let $A$ be an $R$-algebra, and let $I \subseteq A$ be a nilpotent ideal.  Given $R$-algebra homomorphisms $f,g \colon S \to A$ that become equal when composed with $A \to A/I$, we must prove that $f=g$.  Fix $\mathfrak{p}\subset S$.  Then the localizations $A_{\mathfrak{p}} := S_{\mathfrak{p}} \otimes_{S,f} A$ and $S_{\mathfrak{p}} \otimes_{S,g} A$ of $A$ (defined viewing $A$ as an $S$-algebra in the two different ways) are naturally isomorphic, since adjoining the inverse of an $a \in A$ to $A$ automatically makes $a+\epsilon$ invertible for any nilpotent $\epsilon$ (use the geometric series).  Now $f$ and $g$ induce $R$-algebra homomorphisms $f_{\mathfrak{p}},g_{\mathfrak{p}} \colon S_{\mathfrak{p}} \to A_{\mathfrak{p}}$ that become equal when we compose with $A_{\mathfrak{p}} \to A_{\mathfrak{p}}/I A_{\mathfrak{p}}$.  Since $R \to S_{\mathfrak{p}}$ is formally unramified, this means that $f_{\mathfrak{p}} = g_{\mathfrak{p}}$.  In other words, for every $s \in S$, the difference $f(s)-g(s)$ maps to zero in $A_{\mathfrak{p}}$ for every $\mathfrak{p}$.  An element in an $S$-module that becomes $0$ after localizing at each prime ideal of $S$ is $0$, so $f(s)=g(s)$ for all $s$.  So $f=g$.
A: I just thought of a new proof that is for free using modern technology.  It resembles the first step of the other proof, but replacing the sheaf of differentials with the cotangent complex.
In order to see that the map $f:R\to S$ is formally étale, it will be enough to show that the entire cotangent complex vanishes.  The property of a (derived) module being zero can be tested on stalks, so it suffices to show that the stalk of the cotangent complex $\mathbb{L}_{S/R}$ at each prime $\mathfrak{p}\in \operatorname{Spec}(S)$ vanishes, that is to say $$\mathbb{L}_{S/R}\otimes^\mathsf{L}_S S_\mathfrak{p}=0$$ for all primes $\mathfrak{p}\in \operatorname{Spec}(S)$
Suppose $f:R\to S$ such that for every prime $\mathfrak{p}\in \operatorname{Spec}(S),$ we have that the cotangent complex of the composite map $R\to S\to S_\mathfrak{p}$ vanishes, that is to say, $\mathbb{L}_{S_\mathfrak{p}/R}=0$.
But we have a fibre sequence:
$$\mathbb{L}_{S/R}\otimes^\mathsf{L}_S S_\mathfrak{p}\to \mathbb{L}_{S_\mathfrak{p}/R}\to \mathbb{L}_{S_\mathfrak{p}/S}.$$
However, the cotangent complex $\mathbb{L}_{S_\mathfrak{p}/S}$ vanishes because the localization at a prime is a filtered colimit of localizations, which makes it formally étale.  Ergo, we have an equivalence of derived modules
$$\mathbb{L}_{S/R}\otimes^\mathsf{L}_S S_\mathfrak{p}\xrightarrow{\sim} \mathbb{L}_{S_\mathfrak{p}/R},$$
but by assumption, $\mathbb{L}_{S_\mathfrak{p}/R}=0,$ so it follows that
$$\mathbb{L}_{S/R}\otimes^\mathsf{L}_S S_\mathfrak{p}=0,$$
as desired.
A: For just about any ring $R$ and any prime ideal $Q$, taking $S=R_Q$ will give you a counterexample, simply because $R_Q$ usually won't be finitely generated as an $R$-algebra. For example $R=k[x]$ and $Q=(x)$ works.
