Here is a counterexample for $p$ odd.

We will construct a sequence of $\mathbb{F}_p$-algebras $$\mathbb{F}_p[t]=R_0 \subset R_1 \subset R_2 \subset \dotsb$$ such that each $R_i$ has the following properties:

1. For any $f \in R_i$, there exists $j \ge i$ so that $df$ becomes zero in $\Omega^1_{R_j/\mathbb{F}_p}$.
2. $t$ is not a $p$th power in $R_i$.
3. The underlying set of $R_i$ is countable.
4. $R_i$ is $\mathbb{Q}_{\ge 0}$-graded, the degree $0$ piece is $\mathbb{F}_p$, and $\deg t=1$.

Let $R = \varinjlim_i R_i$. Then property 1 implies that $\Omega^1_{R/\mathbb{F}_p}=0$, while property 2 guarantees that $t$ is not a $p$th power in $R$. Properties 3 and 4 will be used to prove properties 1 and 2, respectively.

It suffices to show that if we have a ring $R_i$ satisfying properties 2-4 and a homogeneous element $f\in R_i$ of positive degree, then we can find a ring $R_{i+1} \supset R_i$ so that $R_{i+1}$ also satisfies properties 2-4, and $df=0$ in $\Omega^1_{R_{i+1}/\mathbb{F}_p}$.

The construction will be based on iterated applications of the following observation (which I learned from Johan de Jong): in the ring $$\mathbb{F}_p[x,y,z]/(z^p-xy^p)\,,$$ $d(x^2y^p)=0$ but $x^2 y^p$ is not a $p$th power. Proof: $$d(x^2 y^p)=y^p d(x^2)=2xy^p\,dx=2x\,d(xy^p)=2x\,d(z^p)=0$$ On the other hand, the subring of $p$th powers is generated by $x^p$, $y^p$, $xy^p$, and it is clear that $x^2y^p$ is not in this subring.

So, finally, here is the construction. Choose an integer $n$ so that $\lfloor (n+1)(p+1)/p \rfloor \deg f > 1$. Let $R_{i+1}$ be $R_i[x_1,\dotsc,x_n,y_1,\dotsc,y_n,z]$ modulo the relations \begin{align*} f & = x_1^2 y_1^p \\ x_1 y_1^p & = x_2^2 y_2^p \\ & \dots \\ x_{n-1} y_{n-1}^p & = x_n^2 y_n^p \\ x_n y_n^p & = z^p \,. \end{align*} Give $x_1,\dotsc,x_n,y_1,\dotsc,y_n,z$ any positive grading consistent with the relations. We can show by induction that $d(x_m^2 y_m^p)=0$ for each $m$, so in particular $df=0$. We can also show that for each $m$, $R_i[x_1,\dotsc,x_m,y_1,\dotsc,y_m] \cap R_{i+1}^p$ is generated by $R_i[x_1,\dotsc,x_m,y_1,\dotsc,y_m]^p$ and some elements of the form $(x_m y_m^p)^a$ for $a \ge \lfloor (n-m+1)(p+1)/p \rfloor$, and similarly $R_i \cap R_{i+1}^p$ is generated by $R_i^p$ and some elements of the form $f^a$ for $a \ge \lfloor (n+1)(p+1)/p \rfloor$. In particular, by our choice of $n$, all such elements have degree $>1$. So $t$ cannot become a $p$th power in $R_{i+1}$.

Here is a counterexample for $p$ odd.

We will construct a sequence of $\mathbb{F}_p$-algebras $$\mathbb{F}_p[t]=R_0 \subset R_1 \subset R_2 \subset \dotsb$$ such that each $R_i$ has the following properties:

1. For any $f \in R_i$, there exists $j \ge i$ so that $df$ becomes zero in $\Omega^1_{R_j/\mathbb{F}_p}$.
2. $t$ is not a $p$th power in $R_i$.
3. The underlying set of $R_i$ is countable.
4. $R_i$ is $\mathbb{Q}_{\ge 0}$-graded, the degree $0$ piece is $\mathbb{F}_p$, and $\deg t=1$.

Let $R = \varinjlim_i R_i$. Then property 1 implies that $\Omega^1_{R/\mathbb{F}_p}=0$, while property 2 guarantees that $t$ is not a $p$th power in $R$. Properties 3 and 4 will be used to prove properties 1 and 2, respectively.

It suffices to show that if we have a ring $R_i$ satisfying properties 2-4 and a homogeneous element $f\in R_i$ of positive degree, then we can find a ring $R_{i+1} \supset R_i$ so that $R_{i+1}$ also satisfies properties 2-4, and $df=0$ in $\Omega^1_{R_{i+1}/\mathbb{F}_p}$.

The construction will be based on iterated applications of the following observation (which I learned from Johan de Jong): in the ring $$\mathbb{F}_p[x,y,z]/(z^p-xy^p)\,,$$ $d(x^2y^p)=0$ but $x^2 y^p$ is not a $p$th power. Proof: $$d(x^2 y^p)=y^p d(x^2)=2xy^p\,dx=2x\,d(xy^p)=2x\,d(z^p)=0$$ On the other hand, the subring of $p$th powers is generated by $x^p$, $y^p$, $xy^p$, and it is clear that $x^2y^p$ is not in this subring.

So, finally, here is the construction. Choose an integer $n$ so that $(\lfloor np/(p-1) \rfloor+1) \deg f > 1$. Let $R_{i+1}$ be $R_i[x_1,\dotsc,x_n,y_1,\dotsc,y_n,z]$ modulo the relations \begin{align*} f & = x_1^2 y_1^p \\ x_1 y_1^p & = x_2^2 y_2^p \\ & \dots \\ x_{n-1} y_{n-1}^p & = x_n^2 y_n^p \\ x_n y_n^p & = z^p \,. \end{align*} Give $x_1,\dotsc,x_n,y_1,\dotsc,y_n,z$ any positive grading consistent with the relations. We can show by induction that $d(x_m^2 y_m^p)=0$ for each $m$, so in particular $df=0$. We can also show that for each $m$, $R_i[x_1,\dotsc,x_m,y_1,\dotsc,y_m] \cap R_{i+1}^p$ is generated by $R_i[x_1,\dotsc,x_m,y_1,\dotsc,y_m]^p$ and some elements of the form $(x_m y_m^p)^a$ for $a \ge \lfloor (n-m)p/(p-1) \rfloor+1$ (the $n-m+1$st positive integer not divisible by $p$), and similarly $R_i \cap R_{i+1}^p$ is generated by $R_i^p$ and some elements of the form $f^a$ for $a \ge \lfloor np/(p-1) \rfloor+1$. In particular, by our choice of $n$, all such elements have degree $>1$. So $t$ cannot become a $p$th power in $R_{i+1}$.

Here is a counterexample for $p$ odd.

We will construct a sequence of $\mathbb{F}_p$-algebras $$\mathbb{F}_p[t]=R_0 \subset R_1 \subset R_2 \subset \dotsb$$ having the following properties:

1. For any $i$ and $f \in R_i$, $df$ becomes zero in some $\Omega^1_{R_j/\mathbb{F}_p}$.
2. $t$ is not a $p$th power in any $R_i$.
3. The underlying set of $R_i$ is countable.
4. $R_i$ is $\mathbb{Q}_{\ge 0}$-graded, the degree $0$ piece is $\mathbb{F}_p$, and $\deg t=1$.

Let $R = \varinjlim_i R_i$. Then property 1 implies that $\Omega^1_{R/\mathbb{F}_p}=0$, while property 2 guarantees that $t$ is not a $p$th power in $R$. Properties 3 and 4 will be used to prove properties 1 and 2, respectively.

It suffices to show that if we have a ring $R_i$ satisfying properties 2-4 and a homogeneous element $f\in R_i$ of positive degree, then we can find a ring $R_{i+1} \supset R_i$ so that $R_{i+1}$ also satisfies properties 2-4, and $df=0$ in $\Omega^1_{R_{i+1}/\mathbb{F}_p}$.

The construction will be based on iterated applications of the following observation (which I learned from Johan de Jong): in the ring $$\mathbb{F}_p[x,y,z]/(z^p-xy^p)\,,$$ $d(x^2y^p)=0$ but $x^2 y^p$ is not a $p$th power. Proof: $$d(x^2 y^p)=y^p d(x^2)=2xy^p\,dx=2x\,d(xy^p)=2x\,d(z^p)=0$$ On the other hand, the subring of $p$th powers is generated by $x^p$, $y^p$, $xy^p$, and it is clear that $x^2y^p$ is not in this subring.

So, finally, here is the construction. Choose an integer $n$ so that $\lfloor (n+1)(p+1)/p \rfloor \deg f > 1$. Let $R_{i+1}$ be $R_i[x_1,\dotsc,x_n,y_1,\dotsc,y_n,z]$ modulo the relations \begin{align*} f & = x_1^2 y_1^p \\ x_1 y_1^p & = x_2^2 y_2^p \\ & \dots \\ x_{n-1} y_{n-1}^p & = x_n^2 y_n^p \\ x_n y_n^p & = z^p \,. \end{align*} Give $x_1,\dotsc,x_n,y_1,\dotsc,y_n,z$ any positive grading consistent with the relations. We can show by induction that $d(x_m^2 y_m^p)=0$ for each $m$, so in particular $df=0$. We can also show that for each $m$, $R_i[x_1,\dotsc,x_m,y_1,\dotsc,y_m] \cap R_{i+1}^p$ is generated by $R_i[x_1,\dotsc,x_m,y_1,\dotsc,y_m]^p$ and some elements of the form $(x_m y_m^p)^a$ for $a \ge \lfloor (n-m+1)(p+1)/p \rfloor$, and similarly $R_i \cap R_{i+1}^p$ is generated by $R_i^p$ and some elements of the form $f^a$ for $a \ge \lfloor (n+1)(p+1)/p \rfloor$. In particular, by our choice of $n$, all such elements have degree $>1$. So $t$ cannot become a $p$th power in $R_{i+1}$.

Here is a counterexample for $p$ odd.

We will construct a sequence of $\mathbb{F}_p$-algebras $$\mathbb{F}_p[t]=R_0 \subset R_1 \subset R_2 \subset \dotsb$$ such that each $R_i$ has the following properties:

1. For any $f \in R_i$, there exists $j \ge i$ so that $df$ becomes zero in $\Omega^1_{R_j/\mathbb{F}_p}$.
2. $t$ is not a $p$th power in $R_i$.
3. The underlying set of $R_i$ is countable.
4. $R_i$ is $\mathbb{Q}_{\ge 0}$-graded, the degree $0$ piece is $\mathbb{F}_p$, and $\deg t=1$.

Let $R = \varinjlim_i R_i$. Then property 1 implies that $\Omega^1_{R/\mathbb{F}_p}=0$, while property 2 guarantees that $t$ is not a $p$th power in $R$. Properties 3 and 4 will be used to prove properties 1 and 2, respectively.

It suffices to show that if we have a ring $R_i$ satisfying properties 2-4 and a homogeneous element $f\in R_i$ of positive degree, then we can find a ring $R_{i+1} \supset R_i$ so that $R_{i+1}$ also satisfies properties 2-4, and $df=0$ in $\Omega^1_{R_{i+1}/\mathbb{F}_p}$.

The construction will be based on iterated applications of the following observation (which I learned from Johan de Jong): in the ring $$\mathbb{F}_p[x,y,z]/(z^p-xy^p)\,,$$ $d(x^2y^p)=0$ but $x^2 y^p$ is not a $p$th power. Proof: $$d(x^2 y^p)=y^p d(x^2)=2xy^p\,dx=2x\,d(xy^p)=2x\,d(z^p)=0$$ On the other hand, the subring of $p$th powers is generated by $x^p$, $y^p$, $xy^p$, and it is clear that $x^2y^p$ is not in this subring.

So, finally, here is the construction. Choose an integer $n$ so that $\lfloor (n+1)(p+1)/p \rfloor \deg f > 1$. Let $R_{i+1}$ be $R_i[x_1,\dotsc,x_n,y_1,\dotsc,y_n,z]$ modulo the relations \begin{align*} f & = x_1^2 y_1^p \\ x_1 y_1^p & = x_2^2 y_2^p \\ & \dots \\ x_{n-1} y_{n-1}^p & = x_n^2 y_n^p \\ x_n y_n^p & = z^p \,. \end{align*} Give $x_1,\dotsc,x_n,y_1,\dotsc,y_n,z$ any positive grading consistent with the relations. We can show by induction that $d(x_m^2 y_m^p)=0$ for each $m$, so in particular $df=0$. We can also show that for each $m$, $R_i[x_1,\dotsc,x_m,y_1,\dotsc,y_m] \cap R_{i+1}^p$ is generated by $R_i[x_1,\dotsc,x_m,y_1,\dotsc,y_m]^p$ and some elements of the form $(x_m y_m^p)^a$ for $a \ge \lfloor (n-m+1)(p+1)/p \rfloor$, and similarly $R_i \cap R_{i+1}^p$ is generated by $R_i^p$ and some elements of the form $f^a$ for $a \ge \lfloor (n+1)(p+1)/p \rfloor$. In particular, by our choice of $n$, all such elements have degree $>1$. So $t$ cannot become a $p$th power in $R_{i+1}$.