Following is only a partial answer to the question posted above. More specifically it only attempts to answer $(1)$ of the original question.
First Approach
Definition 1. Let $X$ and $Y$ be two sets and $R\subseteq X\times Y$ be a relation. Let $A\subseteq X$. Then we will define the image of $A$ under the relation $R$, denoted by $R(A)$ as the following set, $$R(A):=\{y\in Y:(x,y)\in R\ \text{for some}\ x\in A\}$$
Let us now try to prove the following lemmas,
Lemma 1. Let $X,Y$ be two sets and $R\subseteq X\times Y$. Let $A\subseteq B\subseteq X$. Then we have, $R(A)\subseteq R(B)$.
Lemma 2. Let $X,Y,Z$ be three sets and $R\subseteq X\times Y$ and $S\subseteq Y\times Z$ are two relations. Then we have, $$(S\circ R)(A)\subseteq S(R(A))$$for all $A\subseteq X$.
Proof. Let us choose $A\subseteq X$. If $(S\circ R)(A)=\emptyset$ then we are done because then $(S\circ R)(A)=\emptyset\subseteq S(R(A))$. So we may assume that $(S\circ R)(A)\ne\emptyset$.
Let $z\in (S\circ R)(A)$. Then there exists some $x\in A$ such that $(x,z)\in S\circ R$. But $(x,z)\in S\circ R$ implies that there exists some $y\in Y$ such that $(x,y)\in R$ and $(y,z)\in S$.
Since $x\in A$ and $(x,y)\in R$ so we can conclude that $y\in R(A)$. Similarly since $y\in R(A)$ and $(y,z)\in S$, we can conclude that $z\in S(R(A))$. Since $z$ was arbitrarily chosen, we have thus shown that, $$(S\circ R)(A)\subseteq S(R(A))$$and furthermore since $A$ was also arbitrarily chosen, we have proved our theorem.
Let us now come to our main definition.
Definition 2. Let $(X,\tau_X)$ and $(Y,\tau_Y)$ be two topological spaces. A relation $R\subseteq X\times Y$ will be said to be continuous iff $R(\overline{A})\subseteq \overline{R(A)}$ for all $A\subseteq X$.
And now the main theorem.
Theorem 1. Let $(X,\tau_X), (Y,\tau_Y)$ and $(Z,\tau_Z)$ be three topological spaces and let $R\subseteq X\times Y$ and $S\subseteq Y\times Z$ be two continuous relations. Then $S\circ R\subseteq X\times Z$ is also continuous.
Proof. Observe that for all sets $A\subseteq X$ we have, \begin{align*}(S\circ R)(\overline{A})&\subseteq S(R(\overline{A}))\\&\subseteq S(\overline{R(A)})&\text{(since}\ R\ \text{is continuous)}\\&\subseteq \overline{S(R(A))}&\text{(since}\ S\ \text{is continuous)}\end{align*}and hence we are done.
Second Approach
Definition 3. Let $X$ and $Y$ be two sets and $R\subseteq X\times Y$ be a relation. Let $B\subseteq Y$. Then we will define the pullback of $B$ under the relation $R$, denoted by $R^{-1}(B)$ as the following set, $$R^{-1}(B):=\{x\in X:(x,y)\in R\ \text{for some}\ y\in B\}$$
Let us now try to prove the following lemmas,
Lemma 3. Let $X,Y$ be two sets and $R\subseteq X\times Y$. Let $A\subseteq B\subseteq Y$. Then we have, $R^{-1}(A)\subseteq R^{-1}(B)$.
Lemma 4. Let $X,Y,Z$ be three sets and $R\subseteq X\times Y$ and $S\subseteq Y\times Z$ are two relations. Then we have, $$(S\circ R)^{-1}(C)\subseteq R^{-1}(S^{-1}(C))$$for all $C\subseteq Z$.
Proof. Let us choose $C\subseteq Z$. If $(S\circ R)^{-1}(C)=\emptyset$ then we are done because then $(S\circ R)^{-1}(C)=\emptyset\subseteq R^{-1}(S^{-1}(C))$. So we may assume that $(S\circ R)^{-1}(C)\ne\emptyset$.
Let $x\in (S\circ R)^{-1}(C)$. Then there exists some $z\in C$ such that $(x,z)\in S\circ R$. But $(x,z)\in S\circ R$ implies that there exists some $y\in Y$ such that $(x,y)\in R$ and $(y,z)\in S$.
Since $z\in C$ and $(y,z)\in S$ so we can conclude that $y\in S^{-1}(C)$. Similarly since $y\in S^{-1}(C)$ and $(x,y)\in R$, we can conclude that $x\in R^{-1}(S^{-1}(C))$. Since $x$ was arbitrarily chosen, we have thus shown that, $$(S\circ R)^{-1}(C)\subseteq R^{-1}(S^{-1}(C))$$and furthermore since $C$ was also arbitrarily chosen, we have proved our theorem.
Let us now come to our main definition.
Definition 4. Let $(X,\tau_X)$ and $(Y,\tau_Y)$ be two topological spaces. A relation $R\subseteq X\times Y$ will be said to be continuous iff $\overline{R^{-1}(B)}\subseteq R^{-1}(\overline{B}) $ for all $B\subseteq Y$.
And now the main theorem.
Theorem 2. Let $(X,\tau_X), (Y,\tau_Y)$ and $(Z,\tau_Z)$ be three topological spaces and let $R\subseteq X\times Y$ and $S\subseteq Y\times Z$ be two continuous relations. Then $S\circ R\subseteq X\times Z$ is also continuous.
Proof. Observe that for all sets $C\subseteq Z$ we have, \begin{align*}\overline{(S\circ R)^{-1}(C)}&\subseteq \overline{R^{-1}(S^{-1}(C))}\\&\subseteq R^{-1}(\overline{S^{-1}(C)})&\text{(since}\ R\ \text{is continuous)}\\&\subseteq R^{-1}(S^{-1}(\overline{C}))&\text{(since}\ S\ \text{is continuous)}\end{align*}and hence we are done.