Given an affine group scheme G over a field of positive characteristic.
Question: Is there a simple criterion for G to be reduced in terms of the neutral Tannakian category of its finite dimensional algebraic representations?
Yes, and one could argue as follows, at least if the group scheme is assumed to be of finite type (so it is an algebraic group), and the base field is perfect. Recall that if $k$ is a field of characteristic zero, then any algebraic group is smooth, and that if $k$ is a perfect field of characteristic $p$, then an algebraic group is reduced if and only if it is smooth.
I'll make some statements using dg-categories, because that language simplifies some statements. Let $k$ be a perfect field, and let $A$ be a $k$-algebra. Then $A$ is smooth (in the usual sense) if and only if $A$ is a perfect object of the (dg-)category $\mathrm{BMod}_A(\mathrm{Mod}_k)$ of $A$-$A$-bimodules in $k$-vector spaces. I don't really have a reference for this, but one could view this result as a jazzed up statement of Serre's regularity criterion. (Since $k$ is perfect, $A$ is smooth if and only if it is regular.) This MathOverflow post has some discussion of such criteria: Smooth dg algebras (and perfect dg modules and compact dg modules).
This implies that if $A$ is a $k$-algebra, then $A$ is smooth if and only if $\mathrm{Mod}_A$ is a dualizable $k$-linear dg-category such that the unit $\eta:\mathrm{Mod}_k\to \mathrm{Mod}_A \otimes_k \mathrm{Mod}_A^\vee$ preserves compact objects. Indeed, the dual of $\mathrm{Mod}_A$ as a $k$-linear dg-category is just the category of modules over the opposite algebra (which is $A$ itself if $A$ is commutative), so $\mathrm{Mod}_A \otimes_k \mathrm{Mod}_A^\vee$ is the category of $A$-$A$-bimodules in $k$-vector spaces. The functor $\eta$ sends the unit $k\in \mathrm{Mod}_k$ to $A$ regarded as a bimodule over itself, so $A$ is smooth by the above discussion.
This gives the desired criterion over a perfect field $k$: an algebraic group $G$ is smooth if and only if the dg-category $\mathrm{Rep}(G)$ is a dualizable object in $k$-linear dg-categories, such that the unit $\eta:\mathrm{Mod}_k\to \mathrm{Rep}(G) \otimes_k \mathrm{Rep}(G)^\vee$ preserves compact objects.
If this is too abstract of a criterion, one could also just stop at the point where we appealed to Serre's homological criterion for regularity. This leads to the statement that an algebraic group $G$ over a perfect field $k$ is smooth if and only if $\mathrm{Rep}(G)$ is of finite global dimension, i.e., $\mathrm{Hom}_{\mathrm{Rep}(G)}(V, W[n])$ vanishes for all finite-dimensional representations $V,W\in \mathrm{Rep}(G)$ and all but finitely many $n$.