I think this works. Suppose we have a ring $R$ and an $R$-module $M$ all of whose localisations are projective and consider $S=S^\ast_RM$, the symmetric algebra on $M$. Then $R \rightarrow S$ is formally smooth precisely when $M$ is projective. Let $\mathfrak p$ be a prime ideal of $S$ and $\mathfrak q$ the prime ideal of $R$ lying below $\mathfrak p$. Then $S_{\mathfrak p}$ is a localisation of $S_{\mathfrak q}=S^\ast_{R_{\mathfrak q}}M_{\mathfrak q}$ which is a formally smooth $R_{\mathfrak q}$-algebra and hence $S_{\mathfrak p}$ is a formally smooth $R$-algebra (localisations being formally étale). To show that the statement is false it thus is enough to find a non-projective $R$-module all of localisation are projective.

If $R$ is a boolean ring (i.e., $r^2=r$ for all $r\in R$) then all its localisations are isomorphic to $\mathbb Z/2$ all of whose modules are projective. Hence it suffices to find a boolean ring $R$ and a non-projective $R$-module. This is easy: Let $S$ be a totally disconnected compact topological space and $s\in S$ a non-isolated point and put $R$ equal to the boolean ring of continuous maps $S \rightarrow \mathbb Z/2$. Evaluation at $s$ gives a ring homomorphism $R \rightarrow \mathbb Z/2$ making $\mathbb Z/2$ an $R$-module. If it were projective there would be a module section $\mathbb Z/2 \rightarrow R$. The image $e$ of $1$ must be the characteristic function of $s$ making that function continuous and $s$ isolated.

I think this works. Suppose we have a ring $R$ and an $R$-module $M$ all of whose localisations are projective and consider $S=S^\ast_RM$, the symmetric algebra on $M$. Then $R \rightarrow S$ is formally smooth precisely when $M$ is projective. Let $\mathfrak p$ be a prime ideal of $S$ and $\mathfrak q$ the prime ideal of $R$ lying below $\mathfrak p$. Then $S_{\mathfrak p}$ is a localisation of $S_{\mathfrak q}=S^\ast_{R_{\mathfrak q}}M_{\mathfrak q}$ which is a formally smooth $R_{\mathfrak q}$-algebra and hence $S_{\mathfrak p}$ is a formally smooth $R$-algebra (localisations being formally étale). To show that the statement is false it thus is enough to find a non-projective $R$-module all of localisation are projective.

If $R$ is a boolean ring (i.e., $r^2=r$ for all $r\in R$) then all its localisations are isomorphic to $\mathbb Z/2$ all of whose modules are projective. Hence it suffices to find a boolean ring $R$ and a non-projective $R$-module. This is easy: Let $S$ be a totally disconnected compact topological space and $s\in S$ a non-isolated point and put $R$ equal to the boolean ring of continuous maps $S \rightarrow \mathbb Z/2$. Evaluation at $s$ gives a ring homomorphism $R \rightarrow \mathbb Z/2$ making $\mathbb Z/2$ an $R$-module. If it were projective there would be a module section $\mathbb Z/2 \rightarrow R$. Let $e$ be the image of $1$. Then for any $f\in R$ we have $fe=f(s)e$. This means that $e$ must be the characteristic function of $s$. Indeed, as $e \mapsto 1$ under evaluation at $s$ we have $e(s)=1$. On the other hand, if $t\neq s$ there is an $f\in R$ with $f(t)=1$ and $f(s)=0$ giving $e(t)=f(t)e(t)=f(s)e(t)=0$. Hence $\{s\}=\{t\in S:e(t)=1\}$ is open so that $s$ is isolated.

Note that we may take for $S$ any infinite compact totally disconnected set as some point of it must be non-isolated. Nice examples are the Cantor set (all of whose points are non-isolated) and the spectrum of $\prod_T\mathbb Z/2$ for any infinite set $T$ (which is the ultrafilter compactification of $T$, any non-principal ultrafilter is non-isolated I think).