I think this works. Suppose we have a ring $R$ and an $R$-module $M$ all of
whose localisations are projective and consider $S=S^\ast_RM$, the symmetric algebra
on $M$. Then $R \rightarrow S$ is formally smooth precisely when $M$ is
projective. Let $\mathfrak p$ be a prime ideal of $S$ and $\mathfrak q$ the
prime ideal of $R$ lying below $\mathfrak p$. Then $S_{\mathfrak p}$ is a
localisation of $S_{\mathfrak q}=S^\ast_{R_{\mathfrak q}}M_{\mathfrak q}$ which is a
formally smooth $R_{\mathfrak q}$-algebra and hence $S_{\mathfrak p}$ is a
formally smooth $R$-algebra (localisations being formally étale). To show that
the statement is false it thus is enough to find a non-projective $R$-module all
of localisation are projective.
If $R$ is a boolean ring (i.e., $r^2=r$ for all $r\in R$) then all its
localisations are isomorphic to $\mathbb Z/2$ all of whose modules are
projective. Hence it suffices to find a boolean ring $R$ and a non-projective
$R$-module. This is easy: Let $S$ be a totally disconnected compact topological
space and $s\in S$ a non-isolated point and put $R$ equal to the boolean ring of
continuous maps $S \rightarrow \mathbb Z/2$. Evaluation at $s$ gives a ring
homomorphism $R \rightarrow \mathbb Z/2$ making $\mathbb Z/2$ an $R$-module. If
it were projective there would be a module section $\mathbb Z/2 \rightarrow
R$. Let $e$ be the image of $1$. Then for any $f\in R$ we have $fe=f(s)e$. This
means that $e$ must be the characteristic function of $s$. Indeed, as $e \mapsto
1$ under evaluation at $s$ we have $e(s)=1$. On the other hand, if $t\neq s$
there is an $f\in R$ with $f(t)=1$ and $f(s)=0$ giving $e(t)=f(t)e(t)=f(s)e(t)=0$.
Hence $\{s\}=\{t\in S:e(t)=1\}$ is open so that $s$ is isolated.
Note that we may take for $S$ any infinite compact totally disconnected set as
some point of it must be non-isolated. Nice examples are the Cantor set (all of whose points are non-isolated) and the
spectrum of $\prod_T\mathbb Z/2$ for any infinite set $T$ (which is the
ultrafilter compactification of $T$, any non-principal ultrafilter is
non-isolated I think).
