Operations on topological spaces which are formed by taking compositions of operations such as union, intersection, complement, and boundary indeed have been studied before.

For instance, Kuratowski's Closure Complement Theorem states that if $X$ is a topological space, then there are at most 14 operations on $X$ formed by taking compositions of closure and complementation and this bound is attained in certain topological spaces. Furthermore, if $X$ is a topological space, then there are at most 7 operations on $X$ formed by taking the composition of interior and closure. These 7 operations take the set $A$ to the following sets:

$A,\textrm{Int}(A),\textrm{Cl}(A),\textrm{Int}(\textrm{Cl}(A)),\textrm{Cl}(\textrm{Int}(A)),\textrm{Int}(\textrm{Cl}(\textrm{Int}(A))),\textrm{Cl}(\textrm{Int}(\textrm{Cl}(A)))$.

The fact that there are no more than these 7 operations follows from the fact that taking interior and taking the closure are idempotent operations, and that we do not get anything new when we alternate between interior and closure too many times. The 14 operations in Kuratowski's Closure Complement Theorem come from the seven operations listed above and their complements.

Things get more complicated when we introduce new operations such as union and intersection however. There is a topological space $X$ and a subset $A\subseteq X$ such that infinitely many sets can be formed from $A$ simply by taking compositions of unions, intersections, complements, interior, and closure.

Therefore, since there are many operations on topological spaces formed by taking the composition of operations such as union, complementation, and closure, it seems like it would be a futile task to give too many of these operations special names, but these operations formed by composing union, complementation, and closure do form an algebraic structure called a closure algebra.

A closure algebra is a Boolean algebra $B$ along with a unary operation $C$ satisfying the identities $x\leq C(x)=C(C(x))$ and $C(x\vee y)=C(x)\vee C(y)$. The operations union, intersection, complementation, and closure on a topological space $X$ make the powerset $P(X)$ into an algebraic structure called a closure algebra. Furthermore, every closure algebra can be embedded into the closure algebra determined by a topological space. Thus, the operations on a topological space formed by taking union, complementations, closure. . . are in a one-to-one correspondence with the elements in a free closure algebra. Furthermore, the equational theory of the variety of closure algebras is decidable, so there is some algorithm to determine whether two of these operations on a topological space are the same operation or not.

frontierof $A$ (though this term is also used as a synonym for boundary). $\endgroup$