There are two generalizations of usual groups: groupoids, where the multiplication operation becomes "partial", and hypergroups, for which the result of multiplying two elements is a probability measure rather than a single element. I am interested in the structure combining both these features (it might have been called "hypergroupoid", but apparently this term is already in use): multiplication is partial and obeys the same rules as in a groupoid, but its result is a probability measure on morphisms with the appropriate source and target objects. Any references?
Yes, the notion of hypergroupoids exits. See the following preprints:
Both the articles are published. The second article defines Haar systems for a locally compact hypergroupoids explains construction of $C^*$-algebra of a hypergroupoid equipped with a Haar system. It also proves the disintegration theorem for hypergroupoids.
(This is more of an extended comment, but contains ideas that may help answer the question)
This might link up with informal ideas that Tim Porter has expressed (only in online forums, not in any formal setting that I know of). He posits the idea that instead of asking for existence fillers in horns for simplicial sets, so as to get Kan complexes, what if there is a probability distribution on the set of possible fillers? To recover your situation, take the nerve of the group considered as a one-object groupoid and this is a reduced Kan complex (reduced = one 0-simplex). That there is a group multiplication means that inner horns have unique fillers. (inverses means unique fillers for outer horns) It may be possible to find a mixture of these two ideas that recovers the definition of a hypergroup.
Then simply relax the condition that your simplicial set is reduced. It is well known that Kan complexes with unique fillers (and poss some other conditions I am too lazy to check) are nerves of groupoids. So this should give the concept you are looking for.
If there were references I'm fairly sure Tim would have mentioned them, but then again, maybe not. If you do find some I'm sure he'd like to hear from you.