If $R$ is a commutative ring and $A$ is a commutative $R$-algebra, we say that an element $x\in A$ is algebraic over $R$ if $x$ is a root of a nonzero polynomial $f \in R[X]$, or equivalently, if the "evaluation at $x$" $R$-algebra homomorphism $R[X]\to A$ is not injective. If we require $f$ to be monic, we get instead the notion of integral elements, and it is well-known that the elements of $A$ that are integral over $R$ form an $R$-subalgebra. It is therefore natural to ask whether the algebraic elements also form a subalgebra.
If $R$ is a field, then algebraic elements coincide with integral elements and therefore form a subalgebra; if $R$ is only an integral domain, we may argue as follows: it suffices to show that for any two algebraic elements $x,y\in A$, the subalgebra $S$ generated by $x$ and $y$ consists solely of algebraic elements, i.e. there is no injective $R$-algebra homomorphism from $R[X]$ to $S$. Consider $K=\text{Frac}(A)$ and the $K$-algebra $K\otimes_R S$ generated by $1\otimes x$ and $1\otimes y$, which are still algebraic because $R\to K$ is injective. Any injective $R[X]\to S$ would induce injective $K[X]\to K\otimes_R S$ due to flatness of $K/R$, which is impossible by the field case.
We may consider a simplified version of this, namely whether an algebra generated by a single algebraic element $x$ consists solely of algebraic elements. For this simplified question, the argument above allows us to reduce to local rings $R$: if $x$ is a root of a nonzero polynomial in $R[X]$, the polynomial remains nonzero in $R_{\mathfrak{m}}[X]$ for some maximal ideal $\mathfrak{m}\subset R$. (But given two elements and two nonzero polynomials they are roots of, there may not exist a single $\mathfrak{m}$ such that both polynomials remain nonzero in $R_{\mathfrak{m}}[X]$.)
There are in fact ways to produce polynomials having $x+y$, $xy$ and $f(x)$ (for $f\in R[X]$) as roots via resultants, given polynomials having $x$ and $y$ as roots, but the problem is that we may get the zero polynomial. For example, zero is what we get if we try to produce a polynomial having $1+X+X^2\in \mathbb{Z}/4\mathbb{Z}[X]/\langle2X^3\rangle$ as a root by computing $\text{res}_X(2X^3,Z-(1+X+X^2))$. However, the following Singular code nonetheless shows that $1+X+X^2$ is a root of $2(X^3+X^2+X+1)$:
ring r=(integer, 4), (x), lp;
map f=r,1+x+x2;
ideal i=2x3;
preimage(r,f,i);
and I've been unable to find a counterexample after experimentations along this line.
I'm therefore asking: for which classes of commutative rings $R$ do we have
(1) every element in an $R$-algebra generated by an algebraic element is algebraic;
(2) algebraic elements form a subalgebra;
(3) idempotency of "algebraic closure" / transitivity of algebraicity: if $x \in A$ is a root of a nonzero polynomial in $A[X]$ with $R$-algebraic coefficients, then $x$ is itself algebraic.
YCor gave a counterexample for (2) (and therefore (3)) in the comments, and darij grinberg pointed out that this question contains a counterexample to (1).