Marshall Hall, in his famous book Theory of Groups, does not always require a binary operation be "welldefined", i.e. an operation is a relation instead of a function (there might be more than one $c$ s.t. $ab=c$). For example, in the discussion of "quasigroup with the inverse property", he use the inverse property to prove that the binary operation is welldefined. This is, I think, different from modern treatment. Could you recommend other references about such treatment?
I am looking at page 7 of Hall's book and I do not see that he is using binary operation in any nonstandard fashion. He defines a "quasigroup with the inverse property" to be a set with a binary product and unary inverse such that $$ a^{1}(ab) = b = (ba)a^{1}. $$ He then wants to show that this is a quasigroup, by which he means a set with a binary operation such that if $ab=c$, then any two of $a$, $b$, $c$ determine the third. This means he must show that $ax=c$ and $yb=c$ have unique solutions, and that is exactly what he does. That $a$ and $b$ determines $c$ is a consequence of his definition of binary operation.
I can see that Hall's wording is slightly misleading, but he is not using binary operation in any sense that differs from what we do now. (In his earlier presentation of the axioms for a group he is quite explicit that a product takes a unique value.)

$\begingroup$ Thanks! I now realize that the second part of discussion is showing the candidate solution is actually a solution. I previously thought it's showing that $c$ is unique. I might be perplexed by Hall's repeated stress on "... is a unique element", giving me the impression that an operation by definition is not necessary "welldefined". In fact he usually put this as an axiom, but not here. Maybe his real intention is to stress that the algebraic structure is closed under the operation. $\endgroup$ – Junyan Xu Oct 7 '11 at 1:21

$\begingroup$ But one can indeed show that $ab$ is unique without assuming it, if we interpret $a^{1}(ab)=b$ as: $(\forall c\in ab)\ a^{1}c=\{b\}$. For if $c,c'\in ab, a^{1}c=a^{1}c'=\{b\}$, so $(a^{1})^{1}b=\{c\}=\{c'\}$, whence $c=c'$. $\endgroup$ – Junyan Xu Oct 7 '11 at 6:56
I haven't checked Hall's book so I'm not entirely sure that this is exactly what he talks about, but there are quite a few generalizations of groups by allowing multivalued operations. I mention some examples in this answer to a question about (canonical) hypergroups.
I think the term under which you can find the most papers is "association scheme". Of course the theory of association schemes is much more general than the theory of groups and is not as understood. For example, an association scheme for which the graph $(X,R_i)$ is connected for every $i$ (using terminology from the wiki link) is called a primitive association scheme. Classifying the commutative primitive association schemes is a hard problem that includes the classification of finite simple groups as a special case. For a general reference see R. Bailey's book Association schemes as well as chapter 12 of Godsil's book "Algebraic Combinatorics".

$\begingroup$ Another source for association schemes is H. Zieschang's webpage blue.utb.edu/zieschang or his book Theory of association schemes. For a first idea what association schemes are about, I'd recommend to read Zieschang's review of Rosemary Bailey's book (see ams.org/journals/bull/20064302/S0273097905010773/…). He mentions there that her book is mostly about commutative association schemes whereas Zieschang's main interest lies in noncommutative ones. $\endgroup$ – Someone Oct 6 '11 at 12:12

$\begingroup$ Thank you for your answer, but it turns out to be my misunderstanding, as commented in Godsil's answer. $\endgroup$ – Junyan Xu Oct 7 '11 at 1:22