EDIT: Don't bother reading my partial solution. Brian Conrad pointed out that an easier way to do what I did is to use the equivalent definition of formally unramified in terms of Kähler differentials. And later on, fpqc posted below a complete solution passed on by Mel Hochster, who got it from Luc Illusie, who got it from ???.
OLD ANSWER:
Here is a half-answer. I'll prove half the conclusion, but on the plus side I'll use only half the hypothesis! Namely, I will prove that if $R \to S_{\mathfrak{p}}$ is formally unramified for all primes $\mathfrak{p} \subset S$, then $R \to S$ is formally unramified.
Let $A$ be an $R$-algebra, and let $I \subseteq A$ be a nilpotent ideal. Given $R$-algebra homomorphisms $f,g \colon S \to A$ that become equal when composed with $A \to A/I$, we must prove that $f=g$. Fix $\mathfrak{p}\subset S$. Then the localizations $A_{\mathfrak{p}} := S_{\mathfrak{p}} \otimes_{S,f} A$ and $S_{\mathfrak{p}} \otimes_{S,g} A$ of $A$ (defined viewing $A$ as an $S$-algebra in the two different ways) are naturally isomorphic, since adjoining the inverse of an $a \in A$ to $A$ automatically makes $a+\epsilon$ invertible for any nilpotent $\epsilon$ (use the geometric series). Now $f$ and $g$ induce $R$-algebra homomorphisms $f_{\mathfrak{p}},g_{\mathfrak{p}} \colon S_{\mathfrak{p}} \to A_{\mathfrak{p}}$ that become equal when we compose with $A_{\mathfrak{p}} \to A_{\mathfrak{p}}/I A_{\mathfrak{p}}$. Since $R \to S_{\mathfrak{p}}$ is formally unramified, this means that $f_{\mathfrak{p}} = g_{\mathfrak{p}}$. In other words, for every $s \in S$, the difference $f(s)-g(s)$ maps to zero in $A_{\mathfrak{p}}$ for every $\mathfrak{p}$. An element in an $S$-module that becomes $0$ after localizing at each prime ideal of $S$ is $0$, so $f(s)=g(s)$ for all $s$. So $f=g$.
