(This is more of a comment than an answer, but too long for the comment box.)
Your property is saying precisely that $[(R\circ R')\upharpoonright\text{dom}(R)]\subseteq R$.
I'm not sure I like your "closure" terminology, since that suggests that you start with a relation, and then close it. But there are in general many relations $R$ that satisfy your property with a given relation $R'$. For example,
- the empty relation $R$ has your property with respect to any $R'$.
- similarly, the full relation $R$ also has this property.
- Also, if $R$ is the transitive closure of $R'$, then this property holds.
So you do not seem to be starting with something and then taking a "closure", but rather asserting that the given relation $R$ is itself already closed under this kind of application with $R'$.