I think this works. Suppose we have a ring $R$ and an $R$-module $M$ all of
whose localisations are projective and consider $S=S^\ast_RM$, the symmetric algebra
on $M$. Then $R \rightarrow S$ is formally smooth precisely when $M$ is
projective. Let $\mathfrak p$ be a prime ideal of $S$ and $\mathfrak q$ the
prime ideal of $R$ lying below $\mathfrak p$. Then $S_{\mathfrak p}$ is a
localisation of $S_{\mathfrak q}=S^\ast_{R_{\mathfrak q}}M_{\mathfrak q}$ which is a
formally smooth $R_{\mathfrak q}$-algebra and hence $S_{\mathfrak p}$ is a
formally smooth $R$-algebra (localisations being formally étale). To show that
the statement is false it thus is enough to find a non-projective $R$-module all
of localisation are projective.

If $R$ is a boolean ring (i.e., $r^2=r$ for all $r\in R$) then all its
localisations are isomorphic to $\mathbb Z/2$ all of whose modules are
projective. Hence it suffices to find a boolean ring $R$ and a non-projective
$R$-module. This is easy: Let $S$ be a totally disconnected compact topological
space and $s\in S$ a non-isolated point and put $R$ equal to the boolean ring of
continuous maps $S \rightarrow \mathbb Z/2$. Evaluation at $s$ gives a ring
homomorphism $R \rightarrow \mathbb Z/2$ making $\mathbb Z/2$ an $R$-module. If
it were projective there would be a module section $\mathbb Z/2 \rightarrow
R$. The image $e$ of $1$ must be the characteristic function of $s$ making that
function continuous and $s$ isolated.