It appears that you yearn to study various first-order theories, but do not want to be constrained by any requirement that your models, or domains of discourse, be sets. There are several ways to take such a proposal.

On the one hand, many mathematicians have yearnings similar to yours, and this has led them to try to use category theory as a theoretical background for their mathematical investigations. Surely this is part of the attraction of category theory, and some promote category theory as a kind of alternative foundation of mathematics (that is, alternative to set theory) for precisely this kind of reason. But you say that you are not especially interested in adopting that view.

Another way to study mathematical structures that are not sets, while keeping a principally set-theoretic background, is to focus on the set-class distinction in set theory. If V is the universe of all sets, we can define certain classes in V, such as { x | φ(x) }, where φ is any property. Such a class is not always a set. For example, the Russell paradox is based on the observations that if the collection R = { x | x ∉ x } were a set, then R ∈ R iff R ∉ R, a contradiction. So R is not a set. But R is still a collection of sorts, and we call it a class. A proper class is a class that is not a set. In ZFC, one can treat classes and proper classes by manipulating their definitions. That is, the classes do not exist as objects within the set-theoretic universe, but rather as definable subcollections of the universe. The intuition is that proper classes are simply too big to be sets. Other proper classes would include the class V itself (consisting of all sets), the class of all ordinals, the class of all cardinals, the class of all groups, all rings, all monoids, etc. Each of these classes is too large to be a set, but each has a perfectly clear definition defining a family of objects.

There are other formalizations of set theory, such as Goedel-Bernays set theory GBC and Kelly-Morse set theory, that allow one to treat classes as objects. In these theories, there are two kinds of objects, sets and classes, and every set is a class, but there are classes that are not sets (such as those I listed above). It turns out that GBC is a conservative extension of ZFC, which means that the assertsions purely about sets that are provable in GBC are exactly the same as the assertions abouts sets that are provable in ZFC. Kelly-Morse, in contrast, is not conservative over ZFC, and it implies, in particular, that their must be set models of ZFC.

Now, the point is that you could study magmas that are proper classes. These would not be sets, but would still exist and could be formally analyzed as mathematical structures in these various set theoretic backgrounds. For example, one magma is simply the set union operation: (a,b) maps to (a U b), defined on all pairs of sets a, b. This magma is not a set, simply because it is much too large. There are innumerably many other such examples.

It appears that you yearn to study various first-order theories, but do not want to be constrained by any requirement that your models, or domains of discourse, be sets. There are several ways to take such a proposal.

On the one hand, many mathematicians have yearnings similar to yours, and this has led them to try to use category theory as a theoretical background for their mathematical investigations. Surely this is part of the attraction of category theory, and some promote category theory as a kind of alternative foundation of mathematics (that is, alternative to set theory) for precisely this kind of reason. But you say that you are not especially interested in adopting that view.

Another way to study mathematical structures that are not sets, while keeping a principally set-theoretic background, is to focus on the set-class distinction in set theory. If V is the universe of all sets, we can define certain classes in V, such as { x | φ(x) }, where φ is any property. Such a class is not always a set. For example, the Russell paradox is based on the observations that if the collection R = { x | x ∉ x } were a set, then R ∈ R iff R ∉ R, a contradiction. So R is not a set. But R is still a collection of sorts, and we call it a class. A proper class is a class that is not a set. In ZFC, one can treat classes and proper classes by manipulating their definitions. That is, the classes do not exist as objects within the set-theoretic universe, but rather as definable subcollections of the universe. The intuition is that proper classes are simply too big to be sets. Other proper classes would include the class V itself (consisting of all sets), the class of all ordinals, the class of all cardinals, the class of all groups, all rings, all monoids, etc. Each of these classes is too large to be a set, but each has a perfectly clear definition defining a family of objects.

There are other formalizations of set theory, such as Goedel-Bernays set theory GBC and Kelly-Morse set theory, that allow one to treat classes as objects. In these theories, there are two kinds of objects, sets and classes, and every set is a class, but there are classes that are not sets (such as those I listed above). It turns out that GBC is a conservative extension of ZFC, which means that the assertions purely about sets that are provable in GBC are exactly the same as the assertions about sets that are provable in ZFC. Kelly-Morse, in contrast, is not conservative over ZFC, and it implies, in particular, that their must be set models of ZFC.

Now, the point is that you could study magmas that are proper classes. These would not be sets, but would still exist and could be formally analyzed as mathematical structures in these various set theoretic backgrounds. For example, one magma is simply the set union operation: (a,b) maps to (a U b), defined on all pairs of sets a, b. This magma is not a set, simply because it is much too large. There are innumerably many other such examples.