Any function $f\colon A\to B$ defines a triple adjunction $f_*\dashv f^{-1}\dashv f_!$ between the powersets $\mathcal{P}(A)$ and $\mathcal{P}(B)$, where
\begin{align*}
f_*(U) &:= \{b\in B\ |\ \text{there exists $a\in U$ such that $f(a)=b$}\}\\
f^{-1}(V) &:= \{a\in A\ |\ f(a)\in V\}\\
f_!(U) &:= \{b\in B\ |\ f^{-1}(b)\subset U\}
\end{align*}
with $U\in\mathcal{P}(A)$ and $V\in\mathcal{P}(B)$. Now, we may define open, closed, and continuous maps using these:
- A map $f$ is **open** if $f_*$ sends opens to opens.
- A map $f$ is **continuous** if $f^{-1}$ sends opens to opens.
- A map $f$ is **closed** if $f_!$ sends opens to opens ([see the proof by მამუკა here](https://mathoverflow.net/a/436539)).
We could now repeat this procedure with relations, although this time the triple adjunction breaks down into two adjunctions: any relation $R\colon A ⇸ B$ defines two adjunctions $R_*\dashv R_{-1}$ and $R^{-1}\dashv R_!$, where
\begin{align*}
R_*(U) &:= \{b\in B\ |\ \text{there exists $a\in U$ such that $b\in R(a)$}\}\\
R_{-1}(V) &:= \{a\in A\ |\ R(a)\subset V\}\\
R^{-1}(V) &:= \{a\in A\ |\ R(a)\cap V\neq\emptyset\}\\
R_!(U) &:= \{b\in B\ |\ R^{-1}(b)\subset U\}
\end{align*}
with $U\in\mathcal{P}(A)$ and $V\in\mathcal{P}(B)$.
> **Note:** A nice fact here is that $R_{-1}=R^{-1}$ iff $R$ is total and functional, i.e. $R^{-1}$ and $R_{-1}$ coincide precisely if $R$ comes from a function.
Mimicking the situation for functions, we could now make the following definitions:
- A relation $R$ is **open** if $R_*$ sends opens to opens.
- A relation $R$ is **strongly continuous** if $R_{-1}$ sends opens to opens.
- A relation $R$ is **weakly continuous** if $R^{-1}$ sends opens to opens.
- A relation $R$ is **closed** if $R_!$ sends opens to opens (A very similar argument to the one given by მამუკა for functions shows that this is the same as asking $R_*$ to send closed sets to closed sets).
There is, however, an issue: continuous maps can be equivalently defined as those $f$ for which $f$ sends closed sets to closed sets, which follows from the equality $A\setminus f^{-1}(V)=f^{-1}(B\setminus V)$. Now, this equality doesn't need to hold for either $R_{-1}$ or $R^{-1}$, as we have
\begin{align*}
R_{-1}(B\setminus V) &= \{a\in A\ |\ R(a)\subset B\setminus V\},\\
A\setminus R_{-1}(V) &= \{a\in A\ |\ R(a)\not\subset V\},\\
R^{-1}(B\setminus V) &= \{a\in A\ |\ R(a)\setminus V\neq\emptyset\},\\
A\setminus R^{-1}(V) &= \{a\in A\ |\ R(a)\cap V=\emptyset\}.
\end{align*}
Considering also relations $R$ for which $R^{-1}$ or $R_{-1}$ send closed sets to closed sets thus leads to a total of four different definitions of continuity for relations using this approach. (**Edit:** the situation turns out to be way better if $R$ is total; see below)
-----
**Edit:** I've since found two references which develop the above ideas further:
- The first one is Clementino–Tholen's _A characterization of the Vietoris topology_ [[PDF]](http://topology.nipissingu.ca/tp/reprints/v22/tp22206.pdf), which develops further the theory of open and closed relations, in particular proving the following theorem:
> **Theorem (Clementino–Tholen).** Let $R\colon X\times Y\to\{0,1\}$ be a relation from $X$ to $Y$, and let $\mathcal{P}^{-}(X)$, $\mathcal{P}^{+}(X)$, and $\mathcal{P}(X)$ denote the lower Vietoris, upper Vietoris, and Vietoris topology on $\mathcal{P}(X)$. The following conditions are equivalent:
>
> 1. The relation $R$ is open.
> 2. The map $R^{-1}\colon\mathcal{P}^{-}(Y)\to\mathcal{P}^{-}(X)$ is continuous.
> 3. The adjunct $R\colon Y\to\mathcal{P}^{-}(X)$ of $R$ is continuous.
> 4. We have $R^{-1}(\overline{V})\subset\overline{R^{-1}(V)}$ for each $V\in\mathcal{P}(Y)$, where $\overline{S}$ denotes the closure of a set $S$.
>
> Similarly, the following conditions are also equivalent:
> 1. The relation $R$ is closed.
> 2. The map $R^{-1}\colon\mathcal{P}^{+}(Y)\to\mathcal{P}^{+}(X)$ is continuous.
> 3. The adjunct $R\colon Y\to\mathcal{P}^{+}(X)$ of $R$ is continuous.
> 4. We have $R_{*}(\overline{U})\supset\overline{R_{*}(U)}$ for each $U\in\mathcal{P}(X)$.
- The second one is Klein–Thompson, _Theory of Correspondences_ [[Link]](https://doi.org/10.1112/blms/17.5.495), which develops the theory of "weakly/strongly continuity" as defined above for total relations (the situation is a little complicated for general relations):
- First, it seems that the definitions of continuity for relations commonly used in practice are instead the following:
- **lower semicontinuity**, called "weak continuity" above;
- **upper semicontinuity**, the property that $R^{-1}$ sends closed sets to closed sets.
- Now, the four definitions situation for continuity of relations described above gets better for total relations: when $R$ is total Proposition 6.3.5 there notes that we have the following equalities:
\begin{align*}
R_{-1}(B\setminus V) &= A\setminus R^{-1}(V),\\
R^{-1}(B\setminus V) &= A\setminus R_{-1}(V).
\end{align*}
As a consequence, $R^{-1}$ preserves opens (resp. closed sets) iff $R_{-1}$ preserves closed sets (resp. opens)!
- Lastly we have the following result (Theorems 7.1.4 and 7.1.7) which gives equivalent conditions for $R$ to be continuous:
> **Theorem (Klein–Thompson).** If $R$ is total, then the following conditions are equivalent:
> 1. The relation $R$ is upper semicontinuous, i.e. $R^{-1}$ sends closed sets to closed sets.
> 2. The adjunct $R\colon X\to\mathcal{P}^{+}(Y)$ of $R$ is continuous.
> 3. The function $R_{-1}$ sends opens to opens.
> 4. For every $x\in X$, every net $(x_n)_{n\in D}$ in $X$ converging to $x$, and every open set $V$ of $Y$ with $f(x)\subset V$, we have $f(x_n)\subset V$ for sufficiently large $n$.
>
> Similarly, the following conditions are equivalent:
> 1. The relation $R$ is lower semicontinuous, i.e. $R^{-1}$ sends opens to opens.
> 2. The adjunct $R\colon X\to\mathcal{P}^{-}(Y)$ of $R$ is continuous.
> 3. The function $R_{-1}$ sends closed sets to closed sets.
> 4. For every $x\in X$, every net $(x_n)_{n\in D}$ in $X$ converging to $x$, and every open set $V$ of $Y$ with $f(x)\cap V\neq\emptyset$, we have $f(x_n)\cap V\neq\emptyset$ for sufficiently large $n$.
Lastly Klein–Thompson also give two results comparing continuity/closedness of relations in the above senses to $R\subset X\times Y$ being a closed set in the product topology:
> **Theorem 7.1.15.** If $Y$ is regular, $R$ is upper semicontinuous, and $R_*(x)$ is closed for each $x\in X$, then $R\subset X\times Y$ is closed with respect to the product topology.
>
> **Theorem 7.1.16.** If $Y$ is compact and $R$ is a closed relation in that $R_*$ maps closed sets to closed sets, then $R\subset X\times Y$ is closed with respect to the product topology.
>
> (Theorem 7.1.16 fails if $Y$ is allowed to be noncompact; see Example 7.1.17 there)
**Some remarks on equivalences relation.**
One reason for us to care about relations being upper/lower semicontinuous, open or closed is because of quotient spaces: when an equivalence relation $\sim$ on a topological space $X$ satisfies some of these properties, we in fact get a bunch of nice facts about the quotient $X/\mathord{\sim}$ being well-behaved. For instance, here's a result from Clementino–Tholen:
> Let $R$ be an equivalence relation on $X$. The following conditions are equivalent:
> 1. The relation $R$ is closed (resp. open)
> 2. The quotient map $\pi\colon X\twoheadrightarrow X/\mathord{\sim}_R$ is closed (resp. open)
> 3. The inclusion map $\iota_{X/\mathord{\sim}_R}\colon X/\mathord{\sim}_R\to\mathcal{P}^{+}(X)$ (resp. to $\mathcal{P}^{-}$(X)) is continuous.
Here are a couple of other results I found:
1. See Daniele Zuddas's answer
2. (Tu _An Introduction to Manifolds_, Theorem 7.7.) If $R$ is an open equivalence relation on $X$, then $X/\mathord{\sim}_R$ is Hausdorff iff $R\subset X\times Y$ is closed in the product topology.
3. (Tu _An Introduction to Manifolds_, Corollary 7.10.) If $R$ is open and $X$ second-countable, then $X/\mathord{\sim}_R$ is second-countable.
4. Daverman's _Decompositions of Manifolds_ has a bunch of other results, too.
**Comparisons with definitions in other answers.** Here's a comparison of the notions above with some of the ones in the other answers:
- **Putting topologies on powersets.** As seen above, continuity, openness and closedness of relations in the above senses correspond to asking the associated functions $X\to\mathcal{P}(Y)$ or $Y\to\mathcal{P}(X)$ to be continuous with respect to the Vietoris topologies. (There are other interesting powerset topologies to consider besides the Vietoris ones, though, like the Fell topology (or maybe the Alexandroff topology; I don't know if it is different from the Vietoris ones))
- **Closedness in the product topology.** Klein–Thompson's Theorems 7.1.15 and 7.1.16 relate closedness of $R$ in the product topology to $R$ being a closed relation. The two notions are distinct in general, however.
- **Eric Wofsey's approach via nets.** As Klein–Thompson's Theorem 7.1.7 shows, a _total_ relation $R$ satisfies Eric's criterion iff it is lower semicontinuous, i.e. iff $R^{-1}$ sends opens to opens.
- **Lehs's upper/lower hemicontinuity.** These are equivalent (I think!) to asking that $R_{-1}$ and $R^{-1}$ preserve opens!