To supplement what Pete says, I'd emphasize that "root systems" have been defined in a variety of ways for a variety of purposes related to Lie theory or to some type of "reflection" group. (For instance, Vinay Deodhar proposed a general notion in the setting of Coxeter groups, which I used in my 1990 book.) The traditional notion arose in the study of semisimple Lie groups and their complexified Lie algebras, where you have finite real reflection groups (Weyl groups = finite crystallographic Coxeter groups). Witt built an early bridge between Coxeter's viewpoint and this kind of Lie theory. Bourbaki's axiomatic treatment encodes all of this efficiently and also includes affine Weyl groups. Eventually many of these notions have been carried over to $p$-adic groups, Kac-Moody theory, etc.

Concerning "complex root systems", these arise in the separate but overlapping theory of finite complex (= unitary) reflection groups. After the classification by Shephard-Todd, Arjeh Cohen systematized somewhat the notion of root system in a 1976 paper "Finite complex reflection groups", freely available here. More recently Michel Broue and his collaborators have used complex reflection groups and associated diagrams (more elaborate versions of Dynkin diagrams) extensively in their study of representations of finite groups of Lie type. There is a useful 2009 Cambridge text by Lehrer and Taylor, Unitary Reflection Groups, with a suitable notion of root system defined at the end of the first chapter.

To supplement what Pete says, I'd emphasize that "root systems" have been defined in a variety of ways for a variety of purposes related to Lie theory or to some type of "reflection" group. (For instance, Vinay Deodhar proposed a general notion in the setting of Coxeter groups, which I used in my 1990 book.) The traditional notion arose in the study of semisimple Lie groups and their complexified Lie algebras, where you have finite real reflection groups (Weyl groups = finite crystallographic Coxeter groups). Witt built an early bridge between Coxeter's viewpoint and this kind of Lie theory. Bourbaki's axiomatic treatment encodes all of this efficiently and also includes affine Weyl groups. Eventually many of these notions have been carried over to $p$-adic groups, Kac-Moody theory, etc.

Concerning "complex root systems", these arise in the separate but overlapping theory of finite complex (= unitary) reflection groups. After the classification by Shephard-Todd, Arjeh Cohen systematized somewhat the notion of root system in a 1976 paper "Finite complex reflection groups", freely available here. More recently Michel Broue and his collaborators have used complex reflection groups and associated diagrams (more elaborate versions of Dynkin diagrams) extensively in their study of representations of finite groups of Lie type. There is a useful 2009 Cambridge text by Lehrer and Taylor, Unitary Reflection Groups, with a suitable notion of root system defined at the end of the first chapter.