This is an attempt to give a more specific and concrete version of Neil Epstein's answer. The initial version was incorrect, and the current version is incomplete.

Let $k$ be a finite field, and fix a sequence of polynomials $u_i(x)\in k[x]$. For $1\leq n<\infty$ put
$$ R_n = k[x,y_0,\dotsc,y_n]/(y_i = x(y_{i+1}+u_i(x))+1), $$
then let $R_\infty$ be the colimit of the rings $R_n$.
Because the relations give $y_i$ in terms of $y_{i+1}$ we just have $R_n=k[x,y_n]$ for $n<\infty$, and this is a UFD. We have $y_i=1\pmod{x}$ for all $i$ so $R_\infty/x=k$. On the other hand, we have $y_i=y_0-i\pmod{x-1}$ for all $i$, so $R_\infty/(x-1)=k[y_0]$. From this it follows that the ideal $(x-1,y_0)$ cannot be principal. Thus, if we can prove that $R_\infty$ is a UFD, then we are done.

One can check that if $f$ is irreducible in $R_n$ and does not lie in $R_n.(x,y_n-1)$ then $f$ remains irreducible in $R_{n+1}$.

Initially I had hoped to take $u_i(x)=0$ for all $i$. However, in this case we find that the elements $p_n=(1-x)y_n-1\in R_n$ satisfy $p_n=x\,p_{n+1}$ in $R_{n+1}$, and it follows that $p_0$ cannot be factored as a product of irreducibles in $R_\infty$.

I still think (by comparison with the details of the example mentioned by Neil Epstein) that it should be possible to produce an example by choosing the polynomials $u_k(x)$ appropriately, possibly as $u_k(x)=x^{m_k}$ for some rapidly increasing sequence $m_k$, perhaps $m_k=k!$. The point is that a certain power series defined in terms of the numbers $m_k$ should be transcendental over $k(x)$. However, I have not understood all the details yet.