Continuous relations? What might it mean for a relation $R\subset X\times Y$ to be continuous, where $X$ and $Y$ are topological spaces? In topology, category theory or in analysis? Is it possible, canonical, useful?
I have a vague idea of the possibility of using continuous relations in science, for example in biochemistry or celestial mechanics, as a less deterministic approach.
I have never seen anything about this topic on internet. I have an opinion of my own, but would like to hear what professional mathematicians think about it.

Can it be adequately defined? And if so, would it possibly be of any
  use at all?


Conditions on the definition:


*

*Composition of continuous relations should be continuous.

*Partial functions, continuous on its domain, should be continuous.

*Given a partial function $f:X\times Y \rightarrow Z$ that is continuous on its domain, 
then $R=\{(x,y)\in X\times Y|f(x,y)=z_0\}$ should be a continuous relation.

 A: Here's a different and quite generic approach: Let $X,Y$ be topological spaces. Then we topologize $\mathcal{P}(Y)$ and say that $R\subseteq X\times Y$ is continuous if and only if the function $f_R: X \to \mathcal{P}(Y)$ defined by $x\mapsto \{y\in Y: (x,y) \in R\}$ is continuous.
As for topologizing $\mathcal{P}(Y)$ you can take the topology generated by $\{\mathcal{S} \subseteq \mathcal{P}(Y): (\bigcup \mathcal{S}) \textrm{ is open in }Y\}$. Possibly other topologies on $\mathcal{P}(Y)$ are more natural.
A: The category of topological spaces and continuous functions does not have canonical notion of a relation. Let me elaborate.
There is only one reasonable 1-categorical notion of a relation: a relation from $A$ to $B$ is a "subobject" of the product $A \times B$. Therefore, if $\mathbb{C}$ is a category, then to define a concept of a relation in $\mathbb{C}$, you have to decide what you mean by a subobject of an object in $\mathbb{C}$.
The most general way to think of a subobject $\phi$ of an object $A$, is to think of $\phi$ as of a logical formula over $A$ (i.e. the "virtual" subobject of $A$ corresponding to formula $\phi$ is given by generalized elements that satisfy the formula $\{a \in A \colon \phi(a)\}$; in the presence of comprehension, such "virtual" subobjects may be materialized in the category, but the point is that we do not need to materialize --- a relation does not have to be representable in the category; it can belong to another world). Therefore, to define subobjects in $\mathbb{C}$, you have to define logic over $\mathbb{C}$. The concept of logic over a category is encapsulated by the concept of fiberwise posetal fibration.
Let us assume that $p \colon \mathbb{U} \rightarrow \mathbb{C}$ is such a fibration over $\mathbb{C}$. A relation $\phi \colon A \nrightarrow B$ in $\mathbb{C}$ corresponds to an object $\phi$ in the fibre of $p$ over $A \times B$. The only problem that remains to solve, is to find a way to compose two relations in such a way that the composition is associative and has neutral elements (i.e. identities). It is not hard to see that to define the composition in the natural way (i.e. ${a (\psi \circ \phi) c} \Leftrightarrow {\exists_{b \in B} {a \phi b} \wedge {b \psi c}}$), our logic $p$ has to have stable cartesian connectives and stable existential quantifiers. Category-theorists call such $p$ a regular logic fibration over $\mathbb{C}$. If you have a regular logic fibration, then you can take its resolution and obtain a 2-posetal category of relations $\mathit{Rel}(p)$ together with a canonical embedding:
$$\mathbb{C} \rightarrow \mathit{Rel}(p)$$
which gives an interpretation of morhpisms from $\mathbb{C}$ as relations in $\mathit{Rel}(p)$.
Now, every (sufficiently complete) category $\mathbb{C}$ has associated one canonical internal logic --- the logic of canonical subobjects (i.e. subobjects associated to monomorphisms $A_0 \rightarrow A$). For example, the canonical internal logic of $\mathbf{Set}$ gives the usual notion of a relation between sets and induces the usual category of relations. It is a good exercise to show that the canonical internal logic of a (finitely complete) category is regular if and only if the category is regular in the usual sense (i.e. it has stable images). Because the category of topological spaces and continuous maps is not regular, there is no canonical notion of a relation between topological spaces. There are three ways to overcome this annoying aspect of topological spaces:


*

*move to a more general category that is regular,

*move to a regular subcategory,

*take a non-canonical logic that is regular over the category of topological spaces.


Which way is the best way depends on your particular applications (I guess this is the reason why your question was closed --- it is completely unclear what you want to achieve). 
BTW, there is a bit more general notion of a relation (and one may encounter it in category theory --- for example in the definition of sheaves over quantales): we can substitute regular logic with regular monoidal logic; i.e. we can weaken cartesian connectives to monoidal connectives and define the composition of $\phi \colon A \nrightarrow B$ with $\psi \colon B \nrightarrow C$ as ${a (\psi \circ \phi) c} \Leftrightarrow {\exists_{b \in B} {a \phi b} \otimes {b \psi c}}$. To be honest, one may imagine even more general notion of a relation, but I do not think it gives us anything useful in our context, so I will refrain from writing about it.
A: Different possibilities to extend continuity to relations:

A relation $X\overset{R}\rightarrow Y$ is continuous if it is upper hemicontinuous and lower hemicontinuous.
Upper hemicontinuous at $a\in X$ if for any open neighbourhood $V$ of $R(a)$ there exists an open neighbourhood $U$ of $a$ such that for all $x\in U$ it holds $R(x)\subset V$.
It seems like upper hemicontinuous is used in game theory for maximizing goal functions.
Lower hemicontinuous at $a\in X$ if for any open set $V$ such that $V\cap R(a)\ne\emptyset$ there exists a neighbourhood $U$ of $a$ such that $V\cap R(x)\ne\emptyset$ for all $x\in U$.

There is a lot of equivalent conditions on continuity for functions that might be considered as candidates for an extension, such as:

*

*$f^{-1}(V)$ is open for all open sets $V\subset Y$

*$f^{-1}(F)$ is closed for all closed sets $F\subset Y$

*$\overline{f^{-1}(M)}\subset f^{-1}(\overline{M})$, for all sets $M\subset Y$

*$f(\overline{L})\subset\overline{f(L)}$, for all sets $L\subset X$

*For all sets $M\subset Y$, it holds that $x\in \overline{f^{-1}(M)}\Rightarrow f(x)\in \overline{M}$
While they are equivalent conditions for functions, they might be non equivalent conditions on relations.

For multivalued functions $\mathcal{F}:X\rightarrow \mathcal M$, hemicontinuity is defined for topologies on $X$ and on a set $\mathcal M$ of subsets. This differs from the question of an eventual extension of continuity from functions to continuity for general relations between topological spaces $R\subset X\times Y$:

*

*first, in the latter case the topology is defined for the points in $Y$,
independent of any induced or otherwise defined topology on a set of
subsets of $Y\!$,

*second, a multivalued function is defined in the whole domain $X$, in contrary to the general case.


About the conditions $(1)$-$(5)$ above: $(2)$ and $(3)$ are equivalent, but non of the others. $(1)$-$(4)$ is satisfied by $X\times Y$, which not in general satisfies $(5)$, see my answer to my question on MATHEMATICS.
A: The discussion here is mostly from the foundations point of view. However, the OP mentioned applications in sciences like celestial mechanics. In fact, for such purposes one might not need general topological spaces. For example, subsets of $\mathbb{R}^n$ may be good enough. Also, sometimes one may get away with only local homeomorphisms as maps. Here is a rather elementary definition of what a reasonable notion of a continuous relation could be generalizing a local homeomorphism:  
A locally homeomorphic relation is a relation $R$ which "locally looks like a homeomorphisms". That is, for any pair of elements $x \in X$ and $y \in Y$, such that $(x, y) \in R,$ there exit neighborhoods $U_x$ and $V_y$ and a homeomorphism between them, whose graph coincides with the restriction of the relation $R$.    
A: This is a remarkable application in topology. It has been given by Mike Freedman in his work about the classification of simply connected closed topological 4-manifolds, which had as a main consequence the topological Poincaré conjecture in dimension 4. Actually, this is one of the many steps in his proof.
Friedman's ball to ball theorem. Let a map $f \colon B^4 \to B^4$ be such that the collection of the inverse sets is null, the singular image is nowhere dense and $f$ is a homeomorphism near the boundary. Then $f$ is approximable by homeomorphisms.
Roughly speaking, the null condition means that the collection of the preimages of $f$ with more than one point, are, a part from finitely many of them, of arbitrary small diameter.
In the proof there is an extraordinary usage of closed relations (that are analogous to continuous functions for compact Hausdorff spaces, as it has been remarked in other answers). The proof starts with a modification of the given map $f$, so that the outcome is not a map, but a relation instead. Then it follows with an infinite construction such that a sequence of relations is defined by induction. The construction is made so that this sequence converges to a homeomorphism close to $f!$
See these notes Chapter 5 for reference, definitions, as well as a proof.
A: Any function $f\colon A\to B$ defines a triple adjunction $f_*\dashv f^{-1}\dashv f_!$ between the powersets $\mathcal{P}(A)$ and $\mathcal{P}(B)$, where
\begin{align*}
    f_*(U)    &:= \{b\in B\ |\ \text{there exists $a\in U$ such that $f(a)=b$}\}\\
    f^{-1}(V) &:= \{a\in A\ |\ f(a)\in V\}\\
    f_!(U)    &:= \{b\in B\ |\ f^{-1}(b)\subset U\}
\end{align*}
with $U\in\mathcal{P}(A)$ and $V\in\mathcal{P}(B)$. Now, we may define open, closed, and continuous maps using these:

*

*A map $f$ is open if $f_*$ sends opens to opens.

*A map $f$ is continuous if $f^{-1}$ sends opens to opens.

*A map $f$ is closed if $f_!$ sends opens to opens (see the proof by მამუკა here).

We could now repeat this procedure with relations, although this time the triple adjunction breaks down into two adjunctions: any relation $R\colon A ⇸ B$ defines two adjunctions $R_*\dashv R_{-1}$ and $R^{-1}\dashv R_!$, where
\begin{align*}
    R_*(U)    &:= \{b\in B\ |\ \text{there exists $a\in U$ such that $b\in R(a)$}\}\\
    R_{-1}(V) &:= \{a\in A\ |\ R(a)\subset V\}\\
    R^{-1}(V) &:= \{a\in A\ |\ R(a)\cap V\neq\emptyset\}\\
    R_!(U)    &:= \{b\in B\ |\ R^{-1}(b)\subset U\}
\end{align*}
with $U\in\mathcal{P}(A)$ and $V\in\mathcal{P}(B)$.

Note: A nice fact here is that $R_{-1}=R^{-1}$ iff $R$ is total and functional, i.e. $R^{-1}$ and $R_{-1}$ coincide precisely if $R$ comes from a function.

Mimicking the situation for functions, we could now make the following definitions:

*

*A relation $R$ is open if $R_*$ sends opens to opens.

*A relation $R$ is strongly continuous if $R_{-1}$ sends opens to opens.

*A relation $R$ is weakly continuous if $R^{-1}$ sends opens to opens.

*A relation $R$ is closed if $R_!$ sends opens to opens (A very similar argument to the one given by მამუკა for functions shows that this is the same as asking $R_*$ to send closed sets to closed sets).

There is, however, an issue: continuous maps can be equivalently defined as those $f$ for which $f$ sends closed sets to closed sets, which follows from the equality $A\setminus f^{-1}(V)=f^{-1}(B\setminus V)$. Now, this equality doesn't need to hold for either $R_{-1}$ or $R^{-1}$, as we have
\begin{align*}
R_{-1}(B\setminus V) &= \{a\in A\ |\ R(a)\subset B\setminus V\},\\
A\setminus R_{-1}(V) &= \{a\in A\ |\ R(a)\not\subset V\},\\
R^{-1}(B\setminus V) &= \{a\in A\ |\ R(a)\setminus V\neq\emptyset\},\\
A\setminus R^{-1}(V) &= \{a\in A\ |\ R(a)\cap V=\emptyset\}.
\end{align*}
Considering also relations $R$ for which $R^{-1}$ or $R_{-1}$ send closed sets to closed sets thus leads to a total of four different definitions of continuity for relations using this approach. (Edit: the situation turns out to be way better if $R$ is total; see below)

Edit: I've since found two references which develop the above ideas further:

*

*The first one is Clementino–Tholen's A characterization of the Vietoris topology [PDF], which develops further the theory of open and closed relations, in particular proving the following theorem:


Theorem (Clementino–Tholen). Let $R\colon X\times Y\to\{0,1\}$ be a relation from $X$ to $Y$, and let $\mathcal{P}^{-}(X)$, $\mathcal{P}^{+}(X)$, and $\mathcal{P}(X)$ denote the lower Vietoris, upper Vietoris, and Vietoris topology on $\mathcal{P}(X)$. The following conditions are equivalent:

*

*The relation $R$ is open.

*The map $R^{-1}\colon\mathcal{P}^{-}(Y)\to\mathcal{P}^{-}(X)$ is continuous.

*The adjunct $R\colon Y\to\mathcal{P}^{-}(X)$ of $R$ is continuous.

*We have $R^{-1}(\overline{V})\subset\overline{R^{-1}(V)}$ for each $V\in\mathcal{P}(Y)$, where $\overline{S}$ denotes the closure of a set $S$.

Similarly, the following conditions are also equivalent:

*

*The relation $R$ is closed.

*The map $R^{-1}\colon\mathcal{P}^{+}(Y)\to\mathcal{P}^{+}(X)$ is continuous.

*The adjunct $R\colon Y\to\mathcal{P}^{+}(X)$ of $R$ is continuous.

*We have $R_{*}(\overline{U})\supset\overline{R_{*}(U)}$ for each $U\in\mathcal{P}(X)$.



*

*The second one is Klein–Thompson, Theory of Correspondences [Link], which develops the theory of "weakly/strongly continuity" as defined above for total relations (the situation is a little complicated for general relations):

*

*First, it seems that the definitions of continuity for relations commonly used in practice are instead the following:

*

*lower semicontinuity, called "weak continuity" above;

*upper semicontinuity, the property that $R^{-1}$ sends closed sets to closed sets.



*Now, the four definitions situation for continuity of relations described above gets better for total relations: when $R$ is total Proposition 6.3.5 there notes that we have the following equalities:
\begin{align*}
    R_{-1}(B\setminus V) &= A\setminus R^{-1}(V),\\
    R^{-1}(B\setminus V) &= A\setminus R_{-1}(V).
\end{align*}
As a consequence, $R^{-1}$ preserves opens (resp. closed sets) iff $R_{-1}$ preserves closed sets (resp. opens)!

*Lastly we have the following result (Theorems 7.1.4 and 7.1.7) which gives equivalent conditions for $R$ to be continuous:




Theorem (Klein–Thompson). If $R$ is total, then the following conditions are equivalent:

*

*The relation $R$ is upper semicontinuous, i.e. $R^{-1}$ sends closed sets to closed sets.

*The adjunct $R\colon X\to\mathcal{P}^{+}(Y)$ of $R$ is continuous.

*The function $R_{-1}$ sends opens to opens.

*For every $x\in X$, every net $(x_n)_{n\in D}$ in $X$ converging to $x$, and every open set $V$ of $Y$ with $f(x)\subset V$, we have $f(x_n)\subset V$ for sufficiently large $n$.

Similarly, the following conditions are equivalent:

*

*The relation $R$ is lower semicontinuous, i.e. $R^{-1}$ sends opens to opens.

*The adjunct $R\colon X\to\mathcal{P}^{-}(Y)$ of $R$ is continuous.

*The function $R_{-1}$ sends closed sets to closed sets.

*For every $x\in X$, every net $(x_n)_{n\in D}$ in $X$ converging to $x$, and every open set $V$ of $Y$ with $f(x)\cap V\neq\emptyset$, we have $f(x_n)\cap V\neq\emptyset$ for sufficiently large $n$.


Lastly Klein–Thompson also give two results comparing continuity/closedness of relations in the above senses to $R\subset X\times Y$ being a closed set in the product topology:

Theorem 7.1.15. If $Y$ is regular, $R$ is upper semicontinuous, and $R_*(x)$ is closed for each $x\in X$, then $R\subset X\times Y$ is closed with respect to the product topology.
Theorem 7.1.16. If $Y$ is compact and $R$ is a closed relation in that $R_*$ maps closed sets to closed sets, then $R\subset X\times Y$ is closed with respect to the product topology.
(Theorem 7.1.16 fails if $Y$ is allowed to be noncompact; see Example 7.1.17 there)

Some remarks on equivalences relation.
One reason for us to care about relations being upper/lower semicontinuous, open or closed is because of quotient spaces: when an equivalence relation $\sim$ on a topological space $X$ satisfies some of these properties, we in fact get a bunch of nice facts about the quotient $X/\mathord{\sim}$ being well-behaved. For instance, here's a result from Clementino–Tholen:

Let $R$ be an equivalence relation on $X$. The following conditions are equivalent:

*

*The relation $R$ is closed (resp. open)

*The quotient map $\pi\colon X\twoheadrightarrow X/\mathord{\sim}_R$ is closed (resp. open)

*The inclusion map $\iota_{X/\mathord{\sim}_R}\colon X/\mathord{\sim}_R\to\mathcal{P}^{+}(X)$ (resp. to $\mathcal{P}^{-}$(X)) is continuous.


Here are a couple of other results I found:

*

*See Daniele Zuddas's answer

*(Tu An Introduction to Manifolds, Theorem 7.7.) If $R$ is an open equivalence relation on $X$, then $X/\mathord{\sim}_R$ is Hausdorff iff $R\subset X\times Y$ is closed in the product topology.

*(Tu An Introduction to Manifolds, Corollary 7.10.) If $R$ is open and $X$ second-countable, then $X/\mathord{\sim}_R$ is second-countable.

*Daverman's Decompositions of Manifolds has a bunch of other results, too.

Comparisons with definitions in other answers. Here's a comparison of the notions above with some of the ones in the other answers:

*

*Putting topologies on powersets. As seen above, continuity, openness and closedness of relations in the above senses correspond to asking the associated functions $X\to\mathcal{P}(Y)$ or $Y\to\mathcal{P}(X)$ to be continuous with respect to the Vietoris topologies. (There are other interesting powerset topologies to consider besides the Vietoris ones, though, like the Fell topology (or maybe the Alexandroff topology; I don't know if it is different from the Vietoris ones))

*Closedness in the product topology. Klein–Thompson's Theorems 7.1.15 and 7.1.16 relate closedness of $R$ in the product topology to $R$ being a closed relation. The two notions are distinct in general, however.

*Eric Wofsey's approach via nets. As Klein–Thompson's Theorem  7.1.7 shows, a total relation $R$ satisfies Eric's criterion iff it is lower semicontinuous, i.e. iff $R^{-1}$ sends opens to opens.

*Lehs's upper/lower hemicontinuity. These are equivalent (I think!) to asking that $R_{-1}$ and $R^{-1}$ preserve opens!

A: Here is an expansion of my comment into an answer which I think is very compelling as the "correct" definition for compact Hausdorff spaces, though I agree with others who have said that for general spaces there may be several competing definitions with different merits.  My argument for this being the right definition is that it is natural in two different ways: it arises naturally by taking the definition of "homomorphism" and modifying it in an obvious way to apply to relations, and it also coincides with the categorical definition of relations as subobjects of the product $X\times Y$.  Furthermore, the coincidence of these two definitions occurs very generally (in particular, in any category monadic over sets).
Let me start by considering this question in different (concrete) categories.  For instance, what might it mean for a relation $R\subseteq G\times H$ to be "homomorphic"?  If you think of a relation as a multivalued function, the following definition seems pretty reasonable: for any $g,g'\in G$, if $h$ is a value of $R(g)$ and $h'$ is a value of $R(g')$, then $hh'$ should be a value of $R(gg')$.  We should also demand that $1$ is a value of $R(1)$ and that if $h$ is a value of $R(g)$, then $h^{-1}$ is a value of $R(g^{-1})$ (demanding these is redundant for functions but not for relations).  It is then easy to check that this is actually equivalent to $R\subseteq G\times H$ being a subgroup of the product group.  This easily generalizes to any other sort of algebraic object: there is an analogous definition of "homomorphic relation", and it is equivalent to being a subobject of the product.
What, then, is the analogue for topological spaces?  Well, if you want to think of a space as a set with some sort of "operations" on it, those operations should be taking limits.  Because limits neither always exist nor are unique in general, there are a few different ways you might define what it means for a relation to preserve limits.  The following is the one I have found to be most natural:
(1)$\,$a relation $R\subseteq X\times Y$ is continuous if whenever $x$ is an accumulation point of a net $(x_a)$ in $X$ and $y_\alpha$ is a value of $R(x_\alpha)$, then there is some accumulation point of $(y_\alpha)$ that is a value of $R(x)$.
Equivalently, we could restrict to universal nets and replace "accumulation point" with "limit" everywhere (however, unlike for functions, it is not equivalent to consider arbitrary nets and replace "accumulation point" with "limit", because there might be values of $R(x)$ that are limits of every universal subnet but no single value that is simultaneously a limit of all of them).
This definition has advantages and disadvantages.  A function is continuous as a relation iff it is continuous in the usual sense and a composition of continuous relations is continuous.  A partial function that is continuous on its domain is continuous as a relation iff its domain is closed.  However, this definition is not symmetric in $X$ and $Y$ (as Joonas Ilmavirta observed, this is a necessary consequence of agreeing with the usual definition on functions).  It also does not coincide with subobjects of $X\times Y$ in the category of topological spaces (which include not only all subspaces of $X\times Y$ but also all subsets equipped with any finer topology).
However, if we restrict to compact Hausdorff spaces, the disadvantages disappear.  Limits of universal nets or ultrafilters are well-defined single-valued operations on compact Hausdorff space, so there is a clear choice for what it means for a relation to be "homomorphic with respect to limits".  A relation between compact Hausdorff spaces is continuous iff it is closed as a subset of $X\times Y$, and thus continuity is symmetric in $X$ and $Y$.  In addition, these continuous relations are also exactly those subsets of $X\times Y$ that are themselves compact Hausdorff spaces, just as in the case of homomorphic relations between algebraic structures.
As a final note, there is a simultaneous generalization of the algebraic case and compact Hausdorff spaces, which is algebras over a monad (compact Hausdorff spaces are the same as algebras over the monad that takes a set to the set of ultrafilters on it, with the structure map of an algebra telling you how to take limits of ultrafilters).  Let $T:\mathrm{Set}\to\mathrm{Set}$ be a monad and let $A$ and $B$ be sets.  Given a relation $R\subseteq A\times B$, we can consider the two projections $A\leftarrow R\to B$ and apply $T$ to get a diagram $TA\leftarrow TR\to TB$.  Let $\tilde{T}R$ be the image of $TR$ in the product $TA\times TB$.  In this way, $T$ naturally extends to a functor $\tilde{T}:\mathrm{Rel}\to\mathrm{Rel}$.
We can now define a "homomorphic relation" between $T$-algebras.  Let $A$ and $B$ be $T$-algebras with structure maps $\mu_A:TA\to A$ and $\mu_B:TB\to B$.  We say a relation $R\subseteq A\times B$ is homomorphic if for any $x\in TA$, if $y$ is a value of $\tilde{T}R(x)$, then $\mu_B(y)$ is a value of $R(\mu_A(x))$.  But this is just saying that $\mu_A\times \mu_B:TA\times TB\to A\times B$ restricts to a map $\tilde{T}R\to R$, and this restriction will then make $R$ itself a $T$-algebra via the composition $TR\to \tilde{T}R\to R$ and a subalgebra of $A\times B$.  Conversely, if $R$ is a subalgebra of $A\times B$, then the structure map $TR\to R$ must factor through $\tilde{T}R$ as a restriction of $\mu_A\times \mu_B$.  Thus homomorphic relations between algebras over a monad always coincide with subalgebras of the product.
A: I have a couple of remarks regarding continuity and some natural constructions of relations:


*

*Unlike functions, relations have no preferred direction.
So if $R\subset X\times Y$ is a relation, its inverse relation $R^{-1}\subset Y\times X$ is an equally valid relation.
Now if we want the concept of a continuous relation to respect this symmetry ($R$ is continuous iff $R^{-1}$ is), we have a significant restriction.
The obvious attempt to define a continuous relation so that the preimage of any open set needs to be (relatively) open leads to a nonsymmetric concept; it is easier to generalise open continuous functions symmetrically.
In fact, a generalization of continuous functions cannot be symmetric (without additional structural assumptions on the spaces) since there are continuous bijections without continuous inverse.

*Let $R\subset X\times Y$ be a relation.
The preimage $R^{-1}Y\subset X$ need not be all of $X$ (unlike for functions).
If we define $R$ to be continuous when the preimage of every open set is open, a partial function obtained by dropping part of a continuous function need not be continuous.
This seems weird (but may be inevitable).
One could also demand that the preimage of any set relatively open in $RX$ (or just open in $Y$ if it seems better) is relatively open in $R^{-1}Y$.

*If $R\subset X\times Y$ and $S\subset Y\times Z$ are relations, their composition $S\circ R=\{(x,z);\exists y\in Y:xRySz\}\subset X\times Z$ is a relation.
If $R$ and $S$ are continuous, it would seem natural to require that $S\circ R$ be continuous as well.
This poses restrictions on the definitions presented in the previous remark; it could happen, for example, that $S^{-1}Z\cap RX\subset Y$ is empty or somehow bad (neither open nor closed).
If we define a continuous relation so that the preimage of an open set must be open, composition preserves continuity, but passing to partial functions does not.
The composition of two (usual/partial/multivalued) functions is again a (usual/partial/multivalued) function, so I think respecting composition is a good idea.
It seems that we can't keep all the good properties of continuous functions and ordinary relations in a theory of continuous relations.
Therefore different applications will probably call for different definitions.
(This vacuously true if there is at most one application.)
A: Since the question is open-ended and basically just seems to be a request for cool ideas, is it okay if I answer a slightly different question? Namely: what is the "right" notion of a measurable relation?
The obvious answer --- take $X$ and $Y$ to be measure spaces and $R$ to be a measurable subset of $X \times Y$ --- is badly behaved. If $X$ is nonatomic then the reflexivity condition for relations on $X$ becomes vacuous, and making sense of transitivity is also problematic.
But there is a good answer! Work with positive measure subsets modulo null sets and assume $X$ and $Y$ are $\sigma$-finite, so we can take joins of arbitrary families of positive measure subsets. Then we characterize measurable relations by saying which pairs of positive measure subsets belong to the relation. The condition is: a measurable relation is a family $R$ of ordered pairs of positive measure subsets of $X$ and $Y$ such that
$$\big(\bigvee A_\alpha, \bigvee B_\alpha\big) \in R\qquad \Leftrightarrow\qquad \mbox{some }(A_\alpha, B_\alpha) \in R,$$
for any families $\{A_\alpha\}$ and $\{B_\alpha\}$ of positive measure subsets of $X$ and $Y$, respectively.
The intuition is that a pair $(A,B)$ belongs to the relation if and only if some point of $A$ is related to some point of $B$.
There is a well-developed theory of measurable relations in this sense. They can be composed, for example. The diagonal relation $\Delta$ is defined by setting $(A,B) \in \Delta$ iff $A \cap B$ is nonnull, and a relation is reflexive if it contains $\Delta$, etc. The details are given in Section 1 of this paper of mine.
Interestingly, as far as I know, there is no good definition of the complement of a measurable relation in this sense.
A: Following is only a partial answer to the question posted above. More specifically it only attempts to answer $(1)$ of the original question.

First Approach

Definition 1. Let $X$ and $Y$ be two sets and $R\subseteq X\times Y$ be a relation. Let $A\subseteq X$. Then we will define the image of $A$ under the relation $R$, denoted by $R(A)$ as the following set, $$R(A):=\{y\in Y:(x,y)\in R\ \text{for some}\ x\in A\}$$

Let us now try to prove the following lemmas,

Lemma 1. Let $X,Y$ be two sets and $R\subseteq X\times Y$. Let $A\subseteq B\subseteq X$. Then we have, $R(A)\subseteq R(B)$.
Lemma 2. Let $X,Y,Z$ be three sets and $R\subseteq X\times Y$ and $S\subseteq Y\times Z$ are two relations. Then we have, $$(S\circ R)(A)\subseteq S(R(A))$$for all $A\subseteq X$.

Proof. Let us choose $A\subseteq X$. If $(S\circ R)(A)=\emptyset$ then we are done because then $(S\circ R)(A)=\emptyset\subseteq S(R(A))$. So we may assume that $(S\circ R)(A)\ne\emptyset$.
Let $z\in (S\circ R)(A)$. Then there exists some $x\in A$ such that $(x,z)\in S\circ R$. But $(x,z)\in S\circ R$ implies that there exists some $y\in Y$ such that $(x,y)\in R$ and $(y,z)\in S$.
Since $x\in A$ and $(x,y)\in R$ so we can conclude that $y\in R(A)$. Similarly since $y\in R(A)$ and $(y,z)\in S$, we can conclude that $z\in S(R(A))$. Since $z$ was arbitrarily chosen, we have thus shown that, $$(S\circ R)(A)\subseteq S(R(A))$$and furthermore since $A$ was also arbitrarily chosen, we have proved our theorem.

Let us now come to our main definition.

Definition 2. Let $(X,\tau_X)$ and $(Y,\tau_Y)$ be two topological spaces. A relation $R\subseteq X\times Y$ will be said to be continuous iff $R(\overline{A})\subseteq \overline{R(A)}$ for all $A\subseteq X$.

And now the main theorem.

Theorem 1. Let $(X,\tau_X), (Y,\tau_Y)$ and $(Z,\tau_Z)$ be three topological spaces and let $R\subseteq X\times Y$ and $S\subseteq Y\times Z$ be two continuous relations. Then $S\circ R\subseteq X\times Z$ is also continuous.

Proof. Observe that for all sets $A\subseteq X$ we have, \begin{align*}(S\circ R)(\overline{A})&\subseteq S(R(\overline{A}))\\&\subseteq S(\overline{R(A)})&\text{(since}\ R\ \text{is continuous)}\\&\subseteq \overline{S(R(A))}&\text{(since}\ S\ \text{is continuous)}\end{align*}and hence we are done.
Second Approach

Definition 3. Let $X$ and $Y$ be two sets and $R\subseteq X\times Y$ be a relation. Let $B\subseteq Y$. Then we will define the pullback of $B$ under the relation $R$, denoted by $R^{-1}(B)$ as the following set, $$R^{-1}(B):=\{x\in X:(x,y)\in R\ \text{for some}\ y\in B\}$$

Let us now try to prove the following lemmas,

Lemma 3. Let $X,Y$ be two sets and $R\subseteq X\times Y$. Let $A\subseteq B\subseteq Y$. Then we have, $R^{-1}(A)\subseteq R^{-1}(B)$.
Lemma 4. Let $X,Y,Z$ be three sets and $R\subseteq X\times Y$ and $S\subseteq Y\times Z$ are two relations. Then we have, $$(S\circ R)^{-1}(C)\subseteq R^{-1}(S^{-1}(C))$$for all $C\subseteq Z$.

Proof. Let us choose $C\subseteq Z$. If $(S\circ R)^{-1}(C)=\emptyset$ then we are done because then $(S\circ R)^{-1}(C)=\emptyset\subseteq R^{-1}(S^{-1}(C))$. So we may assume that $(S\circ R)^{-1}(C)\ne\emptyset$.
Let $x\in (S\circ R)^{-1}(C)$. Then there exists some $z\in C$ such that $(x,z)\in S\circ R$. But $(x,z)\in S\circ R$ implies that there exists some $y\in Y$ such that $(x,y)\in R$ and $(y,z)\in S$.
Since $z\in C$ and $(y,z)\in S$ so we can conclude that $y\in S^{-1}(C)$. Similarly since $y\in S^{-1}(C)$ and $(x,y)\in R$, we can conclude that $x\in R^{-1}(S^{-1}(C))$. Since $x$ was arbitrarily chosen, we have thus shown that, $$(S\circ R)^{-1}(C)\subseteq R^{-1}(S^{-1}(C))$$and furthermore since $C$ was also arbitrarily chosen, we have proved our theorem.

Let us now come to our main definition.

Definition 4. Let $(X,\tau_X)$ and $(Y,\tau_Y)$ be two topological spaces. A relation $R\subseteq X\times Y$ will be said to be continuous iff $\overline{R^{-1}(B)}\subseteq R^{-1}(\overline{B}) $ for all $B\subseteq Y$.

And now the main theorem.

Theorem 2. Let $(X,\tau_X), (Y,\tau_Y)$ and $(Z,\tau_Z)$ be three topological spaces and let $R\subseteq X\times Y$ and $S\subseteq Y\times Z$ be two continuous relations. Then $S\circ R\subseteq X\times Z$ is also continuous.

Proof. Observe that for all sets $C\subseteq Z$ we have, \begin{align*}\overline{(S\circ R)^{-1}(C)}&\subseteq \overline{R^{-1}(S^{-1}(C))}\\&\subseteq R^{-1}(\overline{S^{-1}(C)})&\text{(since}\ R\ \text{is continuous)}\\&\subseteq R^{-1}(S^{-1}(\overline{C}))&\text{(since}\ S\ \text{is continuous)}\end{align*}and hence we are done.
A: I think the key point is to figure out what it means for a relation $R\subset X\times Y$ to "converges" to some point. To achieve this, one could apply the notion of a "convergence space".
So far we have mainly two approaches to define a "convergence space". One approach uses  Moore-Smith's notion of "net", the other one uses Cartan's notion of "filter". It could be shown that these two approaches are essentially the same, if we establish the connection between a net and a filter via taking the "eventual filter" $\mathcal F(\nu)$ for a net $\nu,$ and in this way convert a "net convergence space" into a "filter convergence space". For simplicity I'll only give some details for "filter convergence space".
Following the definition given by R.Beattie and H.P.Butzmann, (which could be found in their treatise Convergence Structures and Applications to Functional Analysis), a convergence space(established on the notion of filters) is a set $X$ endowed with a "convergence structure", which is a family of sets $(\lambda(x))_{x\in X},$ satisfying the following axioms for each $x\in X$:
A1 The members of $\lambda(x)$ are filters on $X;$
A2 If $F\in \lambda(x)$ and $G$ is another filter on $X$ with $F\subset G,$ then $G\in\lambda(X);$
A3 If $F,G\in\lambda(x)$ then $F\cap G\in\lambda(x);$
A4 The principal filter at $x$(i.e., the set of $\left\{A\subset X|\ x\in A\right\}$) belongs to $\lambda(x).$
We say that a filter $F$ on $X$ converges to $x\in X,$ written $F\to x,$ if $F\in\lambda(x).$
Now assume that $X$ and $Y$ areboth endowed with some convergence structures, let $a\in X$ and $b\in Y,$ then we say that the second argument of relation $R$ converges to $b$ as the first argument converges to $a,$ written $\displaystyle\lim_{(x,y)\in R,\ x\to a}y=b,$ if for any filter $F$ on $R,$ we have $\pi_{1}(F)\to a\Longrightarrow \pi_2(F)\to b,$ where $\pi_i:R\to X\ \text{or}\ Y,\ (x,y)\mapsto x\ \text{or}\ y$ are the projection maps.
An equivalent formulation is that, $\displaystyle\lim_{(x,y)\in R,\ x\to a}y=b,$ if for any filter $F$ on $X,$ we have $F\to a\Longrightarrow \left\{S(U)|\ U\in F\right\}\to b,$ where $S(U):=\left\{y\in Y|\ \exists x\in U,\ s.t.\ (x,y)\in R\right\}$ for all subset $U$ of $X.$ 
Once we've established the setting of the convergence of a relation, it's not that difficult to give a definition of "continuous relations". We say a relation $R$ is continuous at point $a\in X,$ if for all $b\in Y,$ if $(a,b)\in R,$ then $\lim_{(x,y)\in R,\ x\to a}y=b.$ It could be seen that the continuity of a function at one point becomes a special case.
To come back to the familiar notion of convergence in a topological space, one can define the convergence structure $(\lambda(x))_{x\in X}$ to be
$$\lambda(x):=\left\{F\ \text{is a filter on $X$}|\ F\supset N(x)\right\},$$
where $N(x)$ is the collection of all neighborhoods of $x,$ which is a filter on $X.$ (BTW, when I say a "neighborhood of $x$" I mean "a subset of $X$ that contains an open set $U$ with $x\in U.$")
Now we can derive from the notion of convergence of a relation  the familiar criterion of the continuity of a function at a point:
$ f:X\to Y$ is continuous at point $x\in X$ if and only if for any neighborhood $U$ of $f(x),$  there is a neighborhood $V$ of $x$ such that $f(V)\subset U.$
A: I'll introduce continuous selection relations or sontinuous for short. I believe them to be useful and having potential for appearing in many topological theorems, say in the topic of selectors.


NOTATION (set theory):
A relation (from $\ X\ $ to $\ Y$) is an ordered triple
$\ (X\,\ Y\ R)\ $ such that $\ R\subseteq X\times Y.$
The composition $\ \circ\ $ of relations $\ (X\,\ Y\ R)\ $ and $\ (Y\ Z\,\ S)\ $ is defined as
$$ (Y\ Z\,\ S)\ \circ\ (X\,\ Y\ R)\,\ =\,\ (X\ Z\ Q) $$
where
$$ Q\ :=\ \{\,(x\ z)\in X\times Z:\ \exists_{\,y\in Y}\,
   (\,(x\ y)\in R\quad\text{and}\quad (y\ z)\in S\,)\} $$
Functions and partial functions are treated as relations too hence one may apply to them the set-theoretical inclusion, etc.
Next, given a relation $\ (X\ Y\ R),\ $ the r-domain
$\ D_R,\ $ and r-codomain $\ C_R\ $ are defined as:
$$ D_R\ :=\ \{x\in X: \exists_{y\in Y}\ (x\ y)\in R\} $$
and
$$ C_R\ :=\ \{y\in Y:\ \exists_{x\in X}\ (x\ y)\in R\}. $$
Finally, a multi-valued function -- I call them the rulers -- is any relation $\ (X\,\ Y\ R)\ $ such that
$$ D_R\ =\ X.$$


Sontinuous relations (topology):
Let $\ (X\ \rho)\ $ and $\ (Y\ \sigma)\ $ be topological spaces. Together with topologies $\ \rho\ $ and $\ \sigma\ $
we consider also the induced topologies as it's standard in topology.
Ruler $\ (X\,\ Y\ R)\ $ is said to be sontinuous
$\ \Leftarrow:\Rightarrow $ there exists a continuous function
(yes, function) $\ s:X\to Y\ $ such that
$\ s\subseteq R.$
A: A function $R\subseteq X\times Y$ is continuous iff for all sets $M\subseteq Y$:
$(1)\quad x\in \overline{R^{-1}(M)}\wedge (x,y)\in R \Rightarrow y\in\overline{M}$, or
$(2)\quad x\in \overline{R^{-1}(M)}\wedge (x,y)\in R \Rightarrow
R(x)\cap\overline{M}\ne\emptyset$
Applied to relations $R\subseteq X\times Y$ the condition $(1)$ implies that $R$ is a (continuous) function if $Y$ is Hausdorff and the condition $(2)$ implies that the maximal relation $X\times Y$ is continuous. Maybe $(1)$ could be of interest for non Hausdorff spaces, but $(2)$ violates my intuition (whatever it is worth) about continuity.
Perhaps one can use conditions between $(1)$ and $(2)$ as
$(3)\quad x\in \overline{R^{-1}(M)}\wedge (x,y)\in R \Rightarrow 
R(x)\cap\overline{M}\ne\emptyset\wedge |R(x)\setminus\overline{M}|\in\mathbb{N}$?
I guess that every algebraic curve is continues in the sense of $(3)$,  but in general not the maximal relation.
Any condition on continuity for functions induces a family of relations. Every open (closed) set $R\subseteq X\times Y$ fulfill the condition that the inverse image of any open (closed) subset of $Y$ is open (closed) in $X$. But for both these conditions the maximal relation is continuous.
