Let $k$ be a field of characteristic $p>0$, $X$ a smooth projective $k$-variety and $Y\subseteq X$ a closed irreducible subvariety. Let $G$ be a connected finite $k$-group scheme acting on $X$.
Does there exist a maximal closed subscheme $T$ of $Y$ stable under the action of $G$?
If $G$ is étale (and not connected), then I think one can define $T:=\bigcap_{g\in G}g(Y)$. In case $T$ exists, is there a similar description for it?
$\newcommand{\cO}{\mathcal{O}}$It seems that your formula for the etale case indeed gives the answer in general, if it is paraphrased in terms of rings of functions.
Consider the coaction map $\Delta:\cO_X\to \cO_X\otimes_k k[G]$ and denote by $I\subset\cO_X$ the ideal sheaf of $Y$. Pick a basis $e_1,\dots, e_n$ for $k[G]$ and let $J$ be the ideal sheaf whose sections on an affine open $U$ are defined by running over all sections $f\in I(U)$, writing $\Delta(f)=\sum f_i\otimes e_i$ and defining $J(U)$ as the $k[U]$-span of all the resulting $f_i$, for all possible $f$. This clearly does not depend on the choice of a basis (in the etale case your formula corresponds to choosing the basis consisting of indicator functions of elements of $G$). The ideal sheaf $J$ contains $I$ because $(\mathrm{id}\otimes e)\circ \Delta=\mathrm{id}$ where $e:k[G]\to k$ is the counit.
Lemma The subscheme $T\subset Y$ defined by $J$ is the maximal closed subscheme of $Y$ preserved by the action of $G$.
Proof. Firstly, the ideal sheaf $J$ satisfies $\Delta(J)\subset J\otimes k[G]$ because the composition $\cO_X\xrightarrow{\Delta}\cO_X\otimes k[G]\xrightarrow{\Delta\otimes\mathrm{id}}\cO_X\otimes k[G]\otimes k[G]$ is equal to $\cO_X\xrightarrow{\Delta}\cO_X\otimes k[G]\xrightarrow{\mathrm{id}\otimes m^*}\cO_X\otimes k[G]\otimes k[G]$ where $m^*$ is the comultiplication on $k[G]$ and the image of $I$ under the first composition contains $\Delta(J)\otimes k[G]$ while the second composition takes $I$ into $J\otimes k[G]\otimes k[G]$. Hence, $T$ is stable under the action of $G$.
Next, suppose that $M\supset I$ is another ideal sheaf such that $\Delta(M)\subset M\otimes k[G]$ (this is equivalent to the zero scheme of $M$ being stable under $G$). Then $\Delta(I)\subset M\otimes k[G]$, so $J\subset M$, by definition. That is, the zero locus of $M$ is contained in $T$.
