Weakly continuous relations have apparently been studied before (see this PlanetMath page), and one nice property of them is that an equivalence relation $\mathord{\sim}$ on a topological space $X$ is weakly continuous iff the projection map $\pi\colon X\twoheadrightarrow X/\mathord{\sim}$ is an open map.

There is, however, an issue: continuous maps can be equivalently defined as those $f$ for which $f$ sends closed sets to closed sets, which follows from the equality $A\setminus f^{-1}(V)=f^{-1}(B\setminus V)$. Now, this equality doesn't need to hold for either $R_{-1}$ or $R^{-1}$, as we have \begin{align*} R_{-1}(B\setminus V) &= \{a\in A\ |\ R(a)\subset B\setminus V\},\\ A\setminus R_{-1}(V) &= \{a\in A\ |\ R(a)\not\subset V\},\\ R^{-1}(B\setminus V) &= \{a\in A\ |\ R(a)\setminus V\neq\emptyset\},\\ A\setminus R^{-1}(V) &= \{a\in A\ |\ R(a)\cap V=\emptyset\}. \end{align*} Considering also relations $R$ for which $R^{-1}$ or $R_{-1}$ send closed sets to closed sets thus leads to a total of four different definitions of continuity for relations using this approach.

There is, however, an issue: continuous maps can be equivalently defined as those $f$ for which $f$ sends closed sets to closed sets, which follows from the equality $A\setminus f^{-1}(V)=f^{-1}(B\setminus V)$. Now, this equality doesn't need to hold for either $R_{-1}$ or $R^{-1}$, as we have \begin{align*} R_{-1}(B\setminus V) &= \{a\in A\ |\ R(a)\subset B\setminus V\},\\ A\setminus R_{-1}(V) &= \{a\in A\ |\ R(a)\not\subset V\},\\ R^{-1}(B\setminus V) &= \{a\in A\ |\ R(a)\setminus V\neq\emptyset\},\\ A\setminus R^{-1}(V) &= \{a\in A\ |\ R(a)\cap V=\emptyset\}. \end{align*} Considering also relations $R$ for which $R^{-1}$ or $R_{-1}$ send closed sets to closed sets thus leads to a total of four different definitions of continuity for relations using this approach. (Edit: the situation turns out to be way better if $R$ is total; see below)

Edit: I've since found two references which develop the above ideas further:

• The first one is Clementino–Tholen's A characterization of the Vietoris topology [PDF], which develops further the theory of open and closed relations, in particular proving the following theorem:

Theorem (Clementino–Tholen). Let $R\colon X\times Y\to\{0,1\}$ be a relation from $X$ to $Y$, and let $\mathcal{P}^{-}(X)$, $\mathcal{P}^{+}(X)$, and $\mathcal{P}(X)$ denote the lower Vietoris, upper Vietoris, and Vietoris topology on $\mathcal{P}(X)$. The following conditions are equivalent:

1. The relation $R$ is open.
2. The map $R^{-1}\colon\mathcal{P}^{-}(Y)\to\mathcal{P}^{-}(X)$ is continuous.
3. The adjunct $R\colon Y\to\mathcal{P}^{-}(X)$ of $R$ is continuous.
4. We have $R^{-1}(\overline{V})\subset\overline{R^{-1}(V)}$ for each $V\in\mathcal{P}(Y)$, where $\overline{S}$ denotes the closure of a set $S$.

Similarly, the following conditions are also equivalent:

1. The relation $R$ is closed.
2. The map $R^{-1}\colon\mathcal{P}^{+}(Y)\to\mathcal{P}^{+}(X)$ is continuous.
3. The adjunct $R\colon Y\to\mathcal{P}^{+}(X)$ of $R$ is continuous.
4. We have $R_{*}(\overline{U})\supset\overline{R_{*}(U)}$ for each $U\in\mathcal{P}(X)$.

• The second one is Klein–Thompson, Theory of Correspondences [Link], which develops the theory of "weakly/strongly continuity" as defined above for total relations (the situation is a little complicated for general relations):
  • First, it seems that the definitions of continuity for relations commonly used in practice are instead the following:
    • lower semicontinuity, called "weak continuity" above;
    • upper semicontinuity, the property that $R^{-1}$ sends closed sets to closed sets.
  • Now, the four definitions situation for continuity of relations described above gets better for total relations: when $R$ is total Proposition 6.3.5 there notes that we have the following equalities: \begin{align*} R_{-1}(B\setminus V) &= A\setminus R^{-1}(V),\\ R^{-1}(B\setminus V) &= A\setminus R_{-1}(V). \end{align*} As a consequence, $R^{-1}$ preserves opens (resp. closed sets) iff $R_{-1}$ preserves closed sets (resp. opens)!
  • Lastly we have the following result (Theorems 7.1.4 and 7.1.7) which gives equivalent conditions for $R$ to be continuous:

Theorem (Klein–Thompson). If $R$ is total, then the following conditions are equivalent:

1. The relation $R$ is upper semicontinuous, i.e. $R^{-1}$ sends closed sets to closed sets.
2. The adjunct $R\colon X\to\mathcal{P}^{+}(Y)$ of $R$ is continuous.
3. The function $R_{-1}$ sends opens to opens.
4. For every $x\in X$, every net $(x_n)_{n\in D}$ in $X$ converging to $x$, and every open set $V$ of $Y$ with $f(x)\subset V$, we have $f(x_n)\subset V$ for sufficiently large $n$.

Similarly, the following conditions are equivalent:

1. The relation $R$ is lower semicontinuous, i.e. $R^{-1}$ sends opens to opens.
2. The adjunct $R\colon X\to\mathcal{P}^{-}(Y)$ of $R$ is continuous.
3. The function $R_{-1}$ sends closed sets to closed sets.
4. For every $x\in X$, every net $(x_n)_{n\in D}$ in $X$ converging to $x$, and every open set $V$ of $Y$ with $f(x)\cap V\neq\emptyset$, we have $f(x_n)\cap V\neq\emptyset$ for sufficiently large $n$.

Lastly Klein–Thompson also give two results comparing continuity/closedness of relations in the above senses to $R\subset X\times Y$ being a closed set in the product topology:

Theorem 7.1.15. If $Y$ is regular, $R$ is upper semicontinuous, and $R_*(x)$ is closed for each $x\in X$, then $R\subset X\times Y$ is closed with respect to the product topology.

Theorem 7.1.16. If $Y$ is compact and $R$ is a closed relation in that $R_*$ maps closed sets to closed sets, then $R\subset X\times Y$ is closed with respect to the product topology.

(Theorem 7.1.16 fails if $Y$ is allowed to be noncompact; see Example 7.1.17 there)

Some remarks on equivalences relation.

One reason for us to care about relations being upper/lower semicontinuous, open or closed is because of quotient spaces: when an equivalence relation $\sim$ on a topological space $X$ satisfies some of these properties, we in fact get a bunch of nice facts about the quotient $X/\mathord{\sim}$ being well-behaved. For instance, here's a result from Clementino–Tholen:

Let $R$ be an equivalence relation on $X$. The following conditions are equivalent:

1. The relation $R$ is closed (resp. open)
2. The quotient map $\pi\colon X\twoheadrightarrow X/\mathord{\sim}_R$ is closed (resp. open)
3. The inclusion map $\iota_{X/\mathord{\sim}_R}\colon X/\mathord{\sim}_R\to\mathcal{P}^{+}(X)$ (resp. to $\mathcal{P}^{-}$(X)) is continuous.

Here are a couple of other results I found:

1. See Daniele Zuddas's answer
2. (Tu An Introduction to Manifolds, Theorem 7.7.) If $R$ is an open equivalence relation on $X$, then $X/\mathord{\sim}_R$ is Hausdorff iff $R\subset X\times Y$ is closed in the product topology.
3. (Tu An Introduction to Manifolds, Corollary 7.10.) If $R$ is open and $X$ second-countable, then $X/\mathord{\sim}_R$ is second-countable.
4. Daverman's Decompositions of Manifolds has a bunch of other results, too.

Comparisons with definitions in other answers. Here's a comparison of the notions above with some of the ones in the other answers:

• Putting topologies on powersets. As seen above, continuity, openness and closedness of relations in the above senses correspond to asking the associated functions $X\to\mathcal{P}(Y)$ or $Y\to\mathcal{P}(X)$ to be continuous with respect to the Vietoris topologies. (There are other interesting powerset topologies to consider besides the Vietoris ones, though, like the Fell topology (or maybe the Alexandroff topology; I don't know if it is different from the Vietoris ones))
• Closedness in the product topology. Klein–Thompson's Theorems 7.1.15 and 7.1.16 relate closedness of $R$ in the product topology to $R$ being a closed relation. The two notions are distinct in general, however.
• Eric Wofsey's approach via nets. As Klein–Thompson's Theorem 7.1.7 shows, a total relation $R$ satisfies Eric's criterion iff it is lower semicontinuous, i.e. iff $R^{-1}$ sends opens to opens.
• Lehs's upper/lower hemicontinuity. These are equivalent (I think!) to asking that $R_{-1}$ and $R^{-1}$ preserve opens!