Here is an expansion of my comment into an answer which I think is very compelling as the "correct" definition for compact Hausdorff spaces, though I agree with others who have said that for general spaces there may be several competing definitions with different merits. My argument for this being the right definition is that it is natural in two different ways: it arises naturally by taking the definition of "homomorphism" and modifying it in an obvious way to apply to relations, and it also coincides with the categorical definition of relations as subobjects of the product $X\times Y$. Furthermore, the coincidence of these two definitions occurs very generally (in particular, in any category monadic over sets).

Let me start by considering this question in different (concrete) categories. For instance, what might it mean for a relation $R\subseteq G\times H$ to be "homomorphic"? If you think of a relation as a multivalued function, the following definition seems pretty reasonable: for any $g,g'\in G$, if $h$ is a value of $R(g)$ and $h'$ is a value of $R(g')$, then $hh'$ should be a value of $R(gg')$. We should also demand that $1$ is a value of $R(1)$ and that if $h$ is a value of $R(g)$, then $h^{-1}$ is a value of $R(g^{-1})$ (demanding these is redundant for functions but not for relations). It is then easy to check that this is actually equivalent to $R\subseteq G\times H$ being a subgroup of the product group. This easily generalizes to any other sort of algebraic object: there is an analogous definition of "homomorphic relation", and it is equivalent to being a subobject of the product.

What, then, is the analogue for topological spaces? Well, if you want to think of a space as a set with some sort of "operations" on it, those operations should be taking limits. Because limits neither always exist nor are unique in general, there are a few different ways you might define what it means for a relation to preserve limits. The following is the one I have found to be most natural:

(1)$\,$a relation $R\subseteq X\times Y$ is continuous if whenever $x$ is an accumulation point of a net $(x_a)$ in $X$ and $y_\alpha$ is a value of $R(x_\alpha)$, then there is some accumulation point of $(y_\alpha)$ that is a value of $R(x)$.

Equivalently, we could restrict to universal nets and replace "accumulation point" with "limit" everywhere (however, unlike for functions, it is not equivalent to consider arbitrary nets and replace "accumulation point" with "limit", because there might be values of $R(x)$ that are limits of every universal subnet but no single value that is simultaneously a limit of all of them).

This definition has advantages and disadvantages. A function is continuous as a relation iff it is continuous in the usual sense and a composition of continuous relations is continuous. A partial function that is continuous on its domain is continuous as a relation iff its domain is closed. However, this definition is not symmetric in $X$ and $Y$ (as Joonas Ilmavirta observed, this is a necessary consequence of agreeing with the usual definition on functions). It also does not coincide with subobjects of $X\times Y$ in the category of topological spaces (which include not only all subspaces of $X\times Y$ but also all subsets equipped with any finer topology).

However, if we restrict to compact Hausdorff spaces, the disadvantages disappear. Limits of universal nets or ultrafilters are well-defined single-valued operations on compact Hausdorff space, so there is a clear choice for what it means for a relation to be "homomorphic with respect to limits". A relation between compact Hausdorff spaces is continuous iff it is closed as a subset of $X\times Y$, and thus continuity is symmetric in $X$ and $Y$. In addition, these continuous relations are also exactly those subsets of $X\times Y$ that are themselves compact Hausdorff spaces, just as in the case of homomorphic relations between algebraic structures.

As a final note, there is a simultaneous generalization of the algebraic case and compact Hausdorff spaces, which is algebras over a monad (compact Hausdorff spaces are the same as algebras over the monad that takes a set to the set of ultrafilters on it, with the structure map of an algebra telling you how to take limits of ultrafilters). Let $T:\mathrm{Set}\to\mathrm{Set}$ be a monad and let $A$ and $B$ be sets. Given a relation $R\subseteq A\times B$, we can consider the two projections $A\leftarrow R\to B$ and apply $T$ to get a diagram $TA\leftarrow TR\to TB$. Let $\tilde{T}R$ be the image of $TR$ in the product $TA\times TB$. In this way, $T$ naturally extends to a functor $\tilde{T}:\mathrm{Rel}\to\mathrm{Rel}$.

We can now define a "homomorphic relation" between $T$-algebras. Let $A$ and $B$ be $T$-algebras with structure maps $\mu_A:TA\to A$ and $\mu_B:TB\to B$. We say a relation $R\subseteq A\times B$ is homomorphic if for any $x\in TA$, if $y$ is a value of $\tilde{T}R(x)$, then $\mu_B(y)$ is a value of $R(\mu_A(x))$. But this is just saying that $\mu_A\times \mu_B:TA\times TB\to A\times B$ restricts to a map $\tilde{T}R\to R$, and this restriction will then make $R$ itself a $T$-algebra via the composition $TR\to \tilde{T}R\to R$ and a subalgebra of $A\times B$. Conversely, if $R$ is a subalgebra of $A\times B$, then the structure map $TR\to R$ must factor through $\tilde{T}R$ as a restriction of $\mu_A\times \mu_B$. Thus homomorphic relations between algebras over a monad always coincide with subalgebras of the product.