What is an algebraic group over a noncommutative ring? Let $R$ be a (noncommutative) ring.  (For me, the words "ring" and "algebra" are isomorphic, and all rings are associative with unit, and usually noncommutative.)  Then I think I know what "linear algebra in characteristic $R$" should be: it should be the study of the category $R\text{-bimod}$ of $(R,R)$-bimodules.  For example, an $R$-algebra on the one hand is a ring $S$ with a ring map $R \to S$.  But this is the same as a ring object in the $R\text{-bimod}$.  When $R$ is a field, we recover the usual linear algebra over $R$; in particular, when $R = \mathbb Z/p$, we recover linear algebra in characteristic $p$.
Suppose that $G$ is an algebraic group (or perhaps I mean "group scheme", and maybe I should say "over $\mathbb Z$"); then my understanding is that for any commutative ring $R$ we have a notion of $G(R)$, which is the group $G$ with coefficients in $R$.  (Probably there are some subtleties and modifications to what I just said.)

My question: What is the right notion of an algebraic group "in characteristic $R$"?

It's certainly a bit funny.  For example, it's reasonable to want $GL(1,R)$ to consist of all invertible elements in $R$.  On the other hand, in $R\text{-bimod}$, the group $\text{Aut}(R,R)$ consists of invertible elements in the center $Z(R)$.
Incidentally, I'm much more interested in how the definitions must be modified to accommodate noncommutativity than in how they must be modified to accommodate non-invertibility.  So I'm happy to set $R = \mathbb H$, the skew field of quaternions.  Or $R = \mathbb K[[x,y]]$, where $\mathbb K$ is a field and $x,y$ are noncommuting formal variables.
 A: Sorry for arriving here so late... I hope somebody will still notice my answer:
A possible definition would be to take a co-group in an appropriate category of noncommutative rings (or algebras), i.e. an object in such a category of noncommutative rings or algebras that represents a functor from this category into a category of groups (see also S. Carnahan's answer).
I know of several papers that follow this approach and study such co-groups, here is what I have been able to find:

*

*Israel Berstein, On co-groups in the category of graded algebras,  Trans. Amer. Math. Soc. 115 (1965), 257–269.


*Dan Voiculescu, Dual algebraic structures on operator algebras related to free products. J. Operator Theory 17 (1987), no. 1, 85–98.


*James J. Zhang, H-algebras. Adv. Math. 89 (1991), no. 2, 144–191.


*George Bergman, Adam Hausknecht,
Co-groups and co-rings in categories of associative rings.
Mathematical Surveys and Monographs, 45. American Mathematical Society, Providence, RI, 1996.


*Benoit Fresse, Cogroups in algebras over an operad are free algebras, Comment. Math. Helv. 73 (1998) 637-676


*Hiroshi Kihara, Cogroups in the category of connected graded algebras whose inverse and antipode coincide, arXiv:1303.7350


*Loïc Foissy, Claudia Malvenuto, and Frederic Patras, B-infinity algebras, their enveloping algebras, and finite spaces. arXiv:1403.7488
Six papers and one book over a span of almost fifty years, citing each other rather sparsely (my list is certainly not complete, though). Compared to the vast literature on algebraic groups and group schemes, this does not seem to be a lot.
Question: Is there an explanation why noncommutative algebraic group theory (in the sense of the OP, maybe one should say noncocommutative?) is getting so little attention? E.g., lack of applications, technical difficulties, lack of interesting results?
A: It seems that you want some notions on noncommutative group scheme,right?
In fact, A.Rosenberg has introduced noncommutative group scheme in his work with Kontsevich
"noncommutative grassmannian and related constructions" (2008). Actually, this work gave a systematically  treatment to the noncommutative grassmannian type space introduced in their early paper noncommutative smooth space and the work of Rosenberg himself on noncommutative spaces and schemes.
More comments: It seems that you want to know the linear algebra over noncommutative ring. I think you need to look at the paper by Gelfand and Retakh on Quasideterminants, I. And the main motivation for the "noncommutative grassmannian and related constructions" is to give a geometric explanation to the work of Gelfand and Retakh.
All of these work are based on functor of point of view.
A: I'd like to add that there is an interesting paper arXiv:math/0701399
 that discusses Lie algebras and groups over noncommutative rings.
A: I'd say an affine algebraic group over $R$ is an $R$-Hopf algebra, that is, a Hopf algebra object in the category of R-R bimodules.  Further than that, it's hard for me to say.
[EDIT: This bit doesn't make any sense.  Ignore it.  I was up until 7am doing Mystery Hunt, so at least I have a good excuse.]  I am pretty suspicious of a definition in terms of the functor of points;  the whole problem with non-commutative geometry is that the points don't capture nearly enough information.
A: There is more than one category of noncommutative spaces of algebraic flavour, hence there is more than one notion of the algebraic group. In affine case, notice that the categorical product noncommutative affine schemes $NAff=Ass^{op}$ is opposite to the free product of the corresponding rings. There are extremely few such schemes, and they correspond to algebra which are very close to the free associative algebras (cf. I. Berstein, On cogroups in the category of graded algebras. Trans. Amer. Math. Soc. 115 (1965), 257–269)-- the example of $NGL_n$ like in Kontsevich-Rosenberg article mentioned by Zhang is just one of the few interesting examples. One can try not to work with categorical product, and work with tensor product like in some approaches to linear quantum groups (B. Parshall and J.Wang, Quantum linear groups. Mem. Amer. Math. Soc. 89 (1991), No. 439, vi+157 pp.), however then some categorical construction do not pass. However if we represent a space by the category of quasicoherent sheaves, then a group scheme is represented by a monoidal category, namely (up to various properness/finiteness conditions) the monoidal product is given by taking the external tensor product of sheaves on the group $G$ what gives a product on $G \times G$ and then one pushes doen this categorical product along the action to $G$. Similar pushdown along the action induces the action of this monoidal category of sheaves on the aprpopriate category of sheaves on the space the group acts on. Then in noncommutative case, we can replace Hopf algebra by its monoidal category of modules, and this category acts on the category of modules over any comodule algebra over that Hopf algebra in a canonical way. This way in the world of categories one indeed has actions of monoidal categories, which are in addition geometrically admissible in the sense explained in my article
Zoran Škoda, Some equivariant constructions in noncommutative algebraic geometry, Georgian Mathematical Journal 16 (2009), No. 1, 183–202, arXiv:0811.4770.
While Kontsevich-Rosenberg treatment of $NGL_n$ is nice functorialy (unfortunately the main part of the work from 1999 is still not a publicly available article) and it was originally motivated by Gel'fand-Retakh quasideterminants this motivation is not fully justified by the results: namely various identities of quasideterminants were not explained as geometric statements about various maps of noncommutative schemes. There is another approach which I develop for a number of years and I hope to be able to finish and write down soon is by taking another version of $NGL_n$ namely the Manin's example of the Hopf envelope of the free matrix bialgebra on $n^2$-generators. This Hopf algebra has infinitely many generators and has interesting structure. There is a geometric quotient which I call a universal noncommutative flag variety. I succeeded to get some of the identities for quasideterminants as geometric statements on that variety. This variety is not a noncommutative scheme, but sort of noncommutative homotopy scheme as the descent is higher descent for Cohn localizations which do not have good flatness properties needed for the usual descent. On the other hand this variety is not represented by a group-valued functor on $NAff$ unlike the noncommutative flag variety of Kontsevich-Rosenberg (which is also glued using Cohn localizations as I was told).
Tomasz Maszczyk has his own approach to noncommutative group schemes (mainly unpublished) which emphasises on the categories of bimodules. But you should talk to him.
A: You seem to be asking two different questions.  The first is, "how do I define the notion of algebraic group over a noncommutative ring?"  The second is, "given an algebraic group (viewed as a functor from commutative rings to groups), how do I evaluate it on noncommutative rings?"  My answers are probably naive, but I don't understand noncommutative geometry.
First question: It should be a functor from rings to groups that preserves finite limits.  You may need more conditions, but this is essentially what you get from the definition of formal groups by removing the "commutative Artinian" condition.
Second question: Evaluate the functor on the center of the ring.  I can't think of a canonical alternative. Edit: Based on the helpful comments, I'd recommend evaluating the functor on the quotient by the two-sided ideal generated by commutators.
