Let $R$ be a (noncommutative) ring. (For me, the words "ring" and "algebra" are isomorphic, and all rings are associative with unit, and usually noncommutative.) Then I think I know what "linear algebra in characteristic $R$" should be: it should be the study of the category $R\text{-bimod}$ of $(R,R)$-bimodules. For example, an $R$-algebra on the one hand is a ring $S$ with a ring map $R \to S$. But this is the same as a ring object in the $R\text{-bimod}$. When $R$ is a field, we recover the usual linear algebra over $R$; in particular, when $R = \mathbb Z/p$, we recover linear algebra in characteristic $p$.

Suppose that $G$ is an algebraic group (or perhaps I mean "group scheme", and maybe I should say "over $\mathbb Z$"); then my understanding is that for any *commutative* ring $R$ we have a notion of $G(R)$, which is the group $G$ with coefficients in $R$. (Probably there are some subtleties and modifications to what I just said.)

My question:What is the right notion of an algebraic group "in characteristic $R$"?

It's certainly a bit funny. For example, it's reasonable to want $GL(1,R)$ to consist of all invertible elements in $R$. On the other hand, in $R\text{-bimod}$, the group $\text{Aut}(R,R)$ consists of invertible elements in the *center* $Z(R)$.

Incidentally, I'm much more interested in how the definitions must be modified to accommodate noncommutativity than in how they must be modified to accommodate non-invertibility. So I'm happy to set $R = \mathbb H$, the skew field of quaternions. Or $R = \mathbb K[[x,y]]$, where $\mathbb K$ is a field and $x,y$ are noncommuting formal variables.