Marshall Hall, in his famous book Theory of Groups, does not always require a binary operation be "welldefined", i.e. an operation is a relation instead of a function (there might be more than one $c$ s.t. $ab=c$). For example, in the discussion of "quasigroup with the inverse property", he use the inverse property to prove that the binary operation is welldefined. This is, I think, different from modern treatment. Could you recommend other references about such treatment?

I am looking at page 7 of Hall's book and I do not see that he is using binary operation in any nonstandard fashion. He defines a "quasigroup with the inverse property" to be a set with a binary product and unary inverse such that $$ a^{1}(ab) = b = (ba)a^{1}. $$ He then wants to show that this is a quasigroup, by which he means a set with a binary operation such that if $ab=c$, then any two of $a$, $b$, $c$ determine the third. This means he must show that $ax=c$ and $yb=c$ have unique solutions, and that is exactly what he does. That $a$ and $b$ determines $c$ is a consequence of his definition of binary operation. I can see that Hall's wording is slightly misleading, but he is not using binary operation in any sense that differs from what we do now. (In his earlier presentation of the axioms for a group he is quite explicit that a product takes a unique value.) 


I haven't checked Hall's book so I'm not entirely sure that this is exactly what he talks about, but there are quite a few generalizations of groups by allowing multivalued operations. I mention some examples in this answer to a question about (canonical) hypergroups. I think the term under which you can find the most papers is "association scheme". Of course the theory of association schemes is much more general than the theory of groups and is not as understood. For example, an association scheme for which the graph $(X,R_i)$ is connected for every $i$ (using terminology from the wiki link) is called a primitive association scheme. Classifying the commutative primitive association schemes is a hard problem that includes the classification of finite simple groups as a special case. For a general reference see R. Bailey's book Association schemes as well as chapter 12 of Godsil's book "Algebraic Combinatorics". 

