$\DeclareMathOperator\GL{GL}$A colleague asked me this question some time ago and here is the answer I sent him.

Let $R$ be a domain with fraction field $K$. Let $A$ be the $R$-algebra underlying the group scheme $\GL_{n,R}$ over $R$.

Suppose given a closed subgroup scheme $H$ of $\GL_{n,K}$. Let $B$ be the underlying $K$-algebra of $H$.
We are then given by assumption a surjection $\phi:A_K\to B$ and an injection $\lambda:A\to A_K$ (given by $a \mapsto a\otimes_K 1$). Let $\mu = \phi\circ\lambda$ and consider the $R$-module $\mu(A)$,
which is a sub-$R$-module of $B$. The following fact can be checked from the definitions: $\mu(A)$ has a natural $R$-algebra structure, compatible with the $R$-algebra structure of B via $\lambda$.
In fact the R-algebra $\mu(A)$ is (by definition) the underlying $R$-algebra of the scheme-theoretic image of $H$ in $\GL_{n,R}$. Furthermore, we have $\mu(A)_K\simeq B$ and
$\mu(A)$ is (clearly) torsion free.
Now consider the Hopf algebra structure of B, which is given by a morphism of $K$-modules $c:B\to B\otimes_K B$. It is easy to see (diagram chasing) that
if $x\in \mu(A)\subseteq B$, then $c(x)$ lies in the image of the natural map $\mu(A)\otimes_R\mu(A)\to B\otimes_K B$. So if the natural
map $\mu(A)\otimes_R\mu(A)\to B\otimes_K B$ is an injection, we obtain a natural map $\mu(A)\to \mu(A)\otimes_R\mu(A)$ and this then
(checking this is elementary) defines a Hopf algebra structure on $\mu(A)$. In particular, if the natural
map $\mu(A)\otimes_R\mu(A)\to B\otimes_K B$ is an injection, then the scheme-theoretic closure of $H$ in $\GL_{n,R}$ is indeed a subgroup scheme, which is reduced if $H$ is reduced
(which is what would happen if $K$ is of characteristic 0).

So the basic condition, which must be satisfied, is that the natural map $\mu(A)\otimes_R\mu(A)\to B\otimes_K B$ is injective. This will be true iff
$\mu(A)\otimes_R\mu(A)$ is torsion free. For example, if $\mu(A)$ happens to be flat over $R$ then $\mu(A)\otimes_R\mu(A)$ will also be flat and thus torsion free.
This is what will happen if $R$ is a Dedekind domain (where torsion free is equivalent to flat) or if $R$ is a valuation ring (where again torsion free is equivalent to flat).
So if we work over either of these two rings, then the scheme-theoretic closure of $H$ in $\GL_{n,R}$ is indeed a subgroup scheme and it is even flat over $R$.

notmean to pass to the reduced structure on the topological closure. $\endgroup$