Here is a counterexample for $p$ odd.
We will construct a sequence of $\mathbb{F}_p$-algebras $$\mathbb{F}_p[t]=R_0 \subset R_1 \subset R_2 \subset \dotsb$$ such that each $R_i$ has the following properties:
- For any $f \in R_i$, there exists $j \ge i$ so that $df$ becomes zero in $\Omega^1_{R_j/\mathbb{F}_p}$.
- $t$ is not a $p$th power in $R_i$.
- The underlying set of $R_i$ is countable.
- $R_i$ is $\mathbb{Q}_{\ge 0}$-graded, the degree $0$ piece is $\mathbb{F}_p$, and $\deg t=1$.
Let $R = \varinjlim_i R_i$. Then property 1 implies that $\Omega^1_{R/\mathbb{F}_p}=0$, while property 2 guarantees that $t$ is not a $p$th power in $R$. Properties 3 and 4 will be used to prove properties 1 and 2, respectively.
It suffices to show that if we have a ring $R_i$ satisfying properties 2-4 and a homogeneous element $f\in R_i$ of positive degree, then we can find a ring $R_{i+1} \supset R_i$ so that $R_{i+1}$ also satisfies properties 2-4, and $df=0$ in $\Omega^1_{R_{i+1}/\mathbb{F}_p}$.
The construction will be based on iterated applications of the following observation (which I learned from Johan de Jong): in the ring $$\mathbb{F}_p[x,y,z]/(z^p-xy^p)\,,$$ $d(x^2y^p)=0$ but $x^2 y^p$ is not a $p$th power. Proof: $$d(x^2 y^p)=y^p d(x^2)=2xy^p\,dx=2x\,d(xy^p)=2x\,d(z^p)=0$$ On the other hand, the subring of $p$th powers is generated by $x^p$, $y^p$, $xy^p$, and it is clear that $x^2y^p$ is not in this subring.
So, finally, here is the construction. Choose an integer $n$ so that $\lfloor (n+1)(p+1)/p \rfloor \deg f > 1$. Let $R_{i+1}$ be $R_i[x_1,\dotsc,x_n,y_1,\dotsc,y_n,z]$ modulo the relations \begin{align*} f & = x_1^2 y_1^p \\ x_1 y_1^p & = x_2^2 y_2^p \\ & \dots \\ x_{n-1} y_{n-1}^p & = x_n^2 y_n^p \\ x_n y_n^p & = z^p \,. \end{align*} Give $x_1,\dotsc,x_n,y_1,\dotsc,y_n,z$ any positive grading consistent with the relations. We can show by induction that $d(x_m^2 y_m^p)=0$ for each $m$, so in particular $df=0$. We can also show that for each $m$, $R_i[x_1,\dotsc,x_m,y_1,\dotsc,y_m] \cap R_{i+1}^p$ is generated by $R_i[x_1,\dotsc,x_m,y_1,\dotsc,y_m]^p$ and some elements of the form $(x_m y_m^p)^a$ for $a \ge \lfloor (n-m+1)(p+1)/p \rfloor$, and similarly $R_i \cap R_{i+1}^p$ is generated by $R_i^p$ and some elements of the form $f^a$ for $a \ge \lfloor (n+1)(p+1)/p \rfloor$. In particular, by our choice of $n$, all such elements have degree $>1$. So $t$ cannot become a $p$th power in $R_{i+1}$.