Here is a counterexample for $p$ odd.

We will construct a sequence of $\mathbb{F}_p$-algebras
$$\mathbb{F}_p[t]=R_0 \subset R_1 \subset R_2 \subset \dotsb$$
such that each $R_i$ has the following properties:

 1. For any $f \in R_i$, there exists $j \ge i$ so that $df$ becomes zero in $\Omega^1_{R_j/\mathbb{F}_p}$.
 2. $t$ is not a $p$th power in $R_i$.
 3. The underlying set of $R_i$ is countable.
 4. $R_i$ is $\mathbb{Q}_{\ge 0}$-graded, the degree $0$ piece is $\mathbb{F}_p$, and $\deg t=1$.

Let $R = \varinjlim_i R_i$.  Then property 1 implies that $\Omega^1_{R/\mathbb{F}_p}=0$, while property 2 guarantees that $t$ is not a $p$th power in $R$.  Properties 3 and 4 will be used to prove properties 1 and 2, respectively.

It suffices to show that if we have a ring $R_i$ satisfying properties 2-4 and a homogeneous element $f\in R_i$ of positive degree, then we can find a ring $R_{i+1} \supset R_i$ so that $R_{i+1}$ also satisfies properties 2-4, and $df=0$ in $\Omega^1_{R_{i+1}/\mathbb{F}_p}$.

The construction will be based on iterated applications of the following observation (which I learned from Johan de Jong): in the ring $$\mathbb{F}_p[x,y,z]/(z^p-xy^p)\,,$$
$d(x^2y^p)=0$ but $x^2 y^p$ is not a $p$th power.  Proof: $$d(x^2 y^p)=y^p d(x^2)=2xy^p\,dx=2x\,d(xy^p)=2x\,d(z^p)=0$$ On the other hand, the subring of $p$th powers is generated by $x^p$, $y^p$, $xy^p$, and it is clear that $x^2y^p$ is not in this subring.

So, finally, here is the construction.  Choose an integer $n$ so that $\lfloor (n+1)(p+1)/p \rfloor \deg f > 1$.  Let $R_{i+1}$ be $R_i[x_1,\dotsc,x_n,y_1,\dotsc,y_n,z]$ modulo the relations
\begin{align*}
f & = x_1^2 y_1^p \\
x_1 y_1^p & = x_2^2 y_2^p \\
& \dots \\
x_{n-1} y_{n-1}^p & = x_n^2 y_n^p \\
x_n y_n^p & = z^p \,.
\end{align*}
Give $x_1,\dotsc,x_n,y_1,\dotsc,y_n,z$ any positive grading consistent with the relations.
We can show by induction that $d(x_m^2 y_m^p)=0$ for each $m$, so in particular $df=0$.  We can also show that for each $m$, $R_i[x_1,\dotsc,x_m,y_1,\dotsc,y_m] \cap R_{i+1}^p$ is generated by $R_i[x_1,\dotsc,x_m,y_1,\dotsc,y_m]^p$ and some elements of the form $(x_m y_m^p)^a$ for $a \ge \lfloor (n-m+1)(p+1)/p \rfloor$, and similarly $R_i \cap R_{i+1}^p$ is generated by $R_i^p$ and some elements of the form $f^a$ for $a \ge \lfloor (n+1)(p+1)/p \rfloor$.  In particular, by our choice of $n$, all such elements have degree $>1$.  So $t$ cannot become a $p$th power in $R_{i+1}$.