The commutative notion of an (associative or not) algebra $A$ over a commutative ring $R$ has two natural generalization to the noncommutative setup, but the one you list with defined left $R$-linearity in both arguments is neither of them; in particular your multiplication does not necessarily induce a map from the tensor product, unless the image of $R$ is in the center. Most useful is the notion of an $R$-ring $A$ (or a ring $A$ over $R$), which is just a monoid in the monoidal category of $R$-bimodules: in other words the multiplication is a map $A\otimes A\to A$ which is left linear in first and right linear in the second factor. If we drop the associativity for the multiplication all works the same way, but I do not know if there is a common name (maybe descriptive like magma internal to the monoidal category of $R$-bimodules; or one may try a rare term nonassociative $R$-ring).
In the commutative case, the mutliplication is both left and right linear in each factor, what is here possible only if $R$ maps into the center of $A$. But that is effectively just the algebra over the center $Z(R)$, so we do not need a new concept(Edit: I erased here one additional nonsense sentence clearly written when tired ;) ). Thus the two useful concepts in the noncommutative case are $R$-rings (possibly nonassociative!) and, well, the subclass with that property: $Z(R)$-algebras. R$ maps into $Z(A)$, deserving the full name of "algebra". There is also a notion of $R$-coring, which is a comonoid in the monoidal category of $R$-bimodules, generalizing the notion of an $R$-coalgebra to a noncommutative ground ring.
Edit: I suggest also this link.
