The discussion here is mostly from the foundations point of view. However, the OP mentioned applications in sciences like celestial mechanics. In fact, for such purposes one might not need general topological spaces. For example, subsets of $R^n$ may be good enough. Also, sometimes one may get away with only local homeomorphisms as maps. Here is a rather elementary definition of what a reasonable notion of a continuous relation could be generalizing a local homeomorphism:
A continuous relation is a relation $R$ which "locally looks like a homeomorphisms". That is, for any pair of elements $x \in X$ and $y \in Y$, there exit neighborhoods $U_x$ and $V_y$ and a homeomorphism between them, whose graph coincides with the restriction of the relation $R$.