In closure spaces (thus, also in topological spaces), one may define the boundary of a set A as the closure of A minus the interior of A. This set is partitioned into "the closure of A minus A" and "A minus the interior of A", that are equal, respectively, to "the boundary of A intersected with the complement of A" and "the boundary of A intersected with A".
Do these two sets partitioning a boundary have a common name? Who studied them?
Q. “Do these two sets partitioning a boundary have a common name?”
A. Yes. They are the rims of $A$ and its complement.
Q. “Who studied them?”
A. Kar-Ping Shum
References
-------------added 7 Oct 2017-----------------
Should have also mentioned: early authors including Kuratowski used the term border to describe this operation. It seems to have faded away at some point (probably before Shum and Yip's 1975 paper). I prefer rim, because in English, the words “border” and “boundary” generally apply to multiple objects at once (the ones being separated by the border or boundary), whereas something's “rim” applies to itself only.
Operations on topological spaces which are formed by taking compositions of operations such as union, intersection, complement, and boundary indeed have been studied before.
For instance, Kuratowski's Closure Complement Theorem states that if $X$ is a topological space, then there are at most 14 operations on $X$ formed by taking compositions of closure and complementation and this bound is attained in certain topological spaces. Furthermore, if $X$ is a topological space, then there are at most 7 operations on $X$ formed by taking the composition of interior and closure. These 7 operations take the set $A$ to the following sets:
$A,\textrm{Int}(A),\textrm{Cl}(A),\textrm{Int}(\textrm{Cl}(A)),\textrm{Cl}(\textrm{Int}(A)),\textrm{Int}(\textrm{Cl}(\textrm{Int}(A))),\textrm{Cl}(\textrm{Int}(\textrm{Cl}(A)))$.
The fact that there are no more than these 7 operations follows from the fact that taking interior and taking the closure are idempotent operations, and that we do not get anything new when we alternate between interior and closure too many times. The 14 operations in Kuratowski's Closure Complement Theorem come from the seven operations listed above and their complements.
Things get more complicated when we introduce new operations such as union and intersection however. There is a topological space $X$ and a subset $A\subseteq X$ such that infinitely many sets can be formed from $A$ simply by taking compositions of unions, intersections, complements, interior, and closure.
Therefore, since there are many operations on topological spaces formed by taking the composition of operations such as union, complementation, and closure, it seems like it would be a futile task to give too many of these operations special names, but these operations formed by composing union, complementation, and closure do form an algebraic structure called a closure algebra.
A closure algebra is a Boolean algebra $B$ along with a unary operation $C$ satisfying the identities $x\leq C(x)=C(C(x))$ and $C(x\vee y)=C(x)\vee C(y)$. The operations union, intersection, complementation, and closure on a topological space $X$ make the powerset $P(X)$ into an algebraic structure called a closure algebra. Furthermore, every closure algebra can be embedded into the closure algebra determined by a topological space. Thus, the operations on a topological space formed by taking union, complementations, closure. . . are in a one-to-one correspondence with the elements in a free closure algebra. Furthermore, the equational theory of the variety of closure algebras is decidable, so there is some algorithm to determine whether two of these operations on a topological space are the same operation or not.
