Does the following relation between sets have a name or any special properties:
$X\bigcirc Y$ iff $X \cap Y = \emptyset$ or $X\subseteq Y$ or $Y\subseteq X$.
Although this is rather basic, it is part of something much grander.
I don't know if there is any better terminology but such families of sets (either disjoint or containing one another) are said to form a Laminar family or a hierarchical system. There is a bijection between Laminar families from a set S and S-rooted forests. If a laminar family has the additional property that each element in S is contained in at least one member of the family then one calls it a complete laminar family. The enumeration of laminar families is open.
I agree that this is a natural and interesting property of a family of sets.
From my (arithmetic-geometric) perspective, the most prominent examples of such families are the disks in an ultrametric space. As Gjergji points out, there is indeed a deep connection to tree-like structures here: see for instance
Already the first paragraph in the introduction underscores my point.
So, although it is certainly not standard, there would be some logic in calling such families "ultrametric".
Good luck with your grandeur. There's plenty of fuel for it in this case, I think.
For the special case where $X$ and $Y$ are intervals of the real line, the complementary relation to this --- i.e. $X \cap Y \neq \varnothing$ but $X \not\subseteq Y$ and $Y \not\subseteq X$ --- is used to define overlap graphs (a.k.a. circle graphs). This is an intersection model of graphs, where vertices are labelled by intervals of the real line, and two vertices are adjacent if and only if those intervals overlap (intersect without nesting).
There is a related concept dealing with separations as opposed to sets. A separation of a set $U$ is an ordered pair $(A,B)$ such that $A$ and $B$ are disjoint subsets of $U$ whose union is $U$. We say that two separations $(A,B)$ and $(C,D)$ are nested if $A \subset C$ or $B \subset D$. Otherwise, we say that the two separations cross. So, if $X$ and $Y$ are both subsets of $U$, then $X$ and $Y$ have your desired property if and only if the separations $(X, U - X)$ and $(Y, U-Y)$ do not cross. I think the term crossing separations is pretty standard, at least in the case of separations of graphs as opposed to separations of sets.