In general if $R$ is a local ring, then its henselization $R^h$ is flat and "algebraic" over $R$, but rarely integral. The intuition is that the henselization is built out of localizations of etale extensions of $R$. Both localizations and etale extensions are flat and "algebraic" in your sense, but localizations are rarely integral.
To make a precise statement, first note that it suffices to find etale ring extensions $R\rightarrow R'\rightarrow S$ with $R$ normal, $S/R$ an etale local homomorphism of local rings, and $S/R'$ not integral.
Indeed, if we have found $R,R',S$ as above, then $S/R'$ non-integral implies $S/R$ non-integral. Moreover, etale morphisms preserve normality, so $S$ must be normal, hence a domain since $S$ is local so $\text{Spec }S$ is connected. Finally, etale ring maps are locally standard etale, so every element of $S$ is "algebraic" over $R$.
Here's a sketch of how to produce an example: Start with a connected normal scheme $Y$ over $\mathbb{C}$ and a finite flat map $f : X\rightarrow Y$ with $X$ irreducible. In characteristic 0, $f$ is generically etale, so let $y\in Y$ be a point above which $f$ is etale. Let $R := \mathcal{O}_{Y,y}$, let $X_R := X\times_Y\text{Spec }\mathcal{O}_{Y,y}$, and write $X_R = \text{Spec }R'$. Then $R'/R$ is finite etale. On the other hand, $R'$ is connected, since the generic point of $X$ maps to the generic point of $Y$ which lies in $\text{Spec }R$, so the generic point of $X$ lies in $X_R$ and specializes to every closed point of $X_R$, so $R'$ is a normal domain. However, if $X$ has multiple closed points lying over $y$, then $R'$ is not local, in which case let $\mathfrak{m}$ be a maximal ideal of $R'$, and let $S := R'_{\mathfrak{m}}$. Then $S/R$ becomes an etale local homomorphism of local rings with $S/R'$ non-integral, as desired.
For an explicit example, you can take $Y = \text{Spec }\mathbb{C}[y]$, and $$X := \text{Spec }\mathbb{C}[x,y]/(y^2-(x-a)(x-b)(x-c))$$ where $a,b,c\in\mathbb{C}$ are distinct. Then take $y\in Y$ to be the point $(y = 0)$. In particular, this shows that the henselization of $\mathbb{C}[y]_{(y)}$ also satisfies your conditions.