Let $k$ be an algebraically closed field of characteristic $p>0$. All the examples of non-smooth algebraic group schemes over $k$ that I have seen (apart from "artificial" examples; see below) have been given by presentations with at least one defining relation of degree a positive power of $p$. Here are the examples I have looked at:
- Frobenius kernels.
- Some automorphism schemes of algebras. The paper "Non-reduced automorphism schemes" by Geiss and Voigt has an interesting example in Section 2, which is given by a presentation including some relations of degree 2. The authors state that this group scheme is not reduced if and only if $p=2$. They also mention that if $G$ is a finite $p$-group, then the group scheme $\mathrm{Aut}(k[G])$ is not reduced.
- Results by Sopkina, "Classification of all connected subgroup schemes of a reductive group containing a split maximal torus", imply, if I have understood things correctly, that every non-smooth subgroup scheme of a reductive group over $k$ (or at least of $\mathrm{GL}_{n}$) containing a maximal torus, has a presentation including a $p$-power relation.
- Non-smooth centralisers in classical groups in not very good characteristic.
On the other hand, it is easy to produce "artificial" examples of a non-smooth group scheme in, for example, char 3 with a presentation involving only quadratic relations. Namely, take $k$ of char 3 and $\alpha_{3}=\mathrm{Spec}\, k[x]/(x^{3})$. Since $k[x]/(x^{3})$ is isomorphic to $k[x,y]/(xy,x-y^{2})$ as $k$-algebras, we can transport the Hopf algebra structure from the former to the latter.
Question. Is there a non-smooth algebraic group scheme $G$ over $k$, and an embedding $G\rightarrow\mathbb{A}^{n}$ of $G$ as a closed subscheme of affine $n$-space, with $n$ minimal, such that every defining relation of $G$ in this embedding is of degree strictly less than $p$? (Even the case $p=3$ would be interesting.)
Using explicit equations and a minimal embedding into affine space may seem a bit unnatural, but results of Kollár and Jelonek (see this previous MO question) - which boil down to estimating degrees and Bézout's theorem - imply that if we take $p>d^{n}$, where $d$ is the maximal degree of a relation, then $p$ does not divide the nilpotency index of any element in the coordinate algebra of $G$, so a proof of Cartier's theorem can be carried through in this case. Hence an example as in the above question must have $d<p<d^n$.