The discussion here is mostly from the foundations point of view. However, the OP mentioned applications in sciences like celestial mechanics. In fact, for such purposes one might not need general topological spaces. For example, subsets of $R^n$ may be good enough. Also, sometimes one may get away with only local homeomorphisms as maps. Here is a rather elementary definition of what a reasonable notion of a continuous relation could be generalizing a local homeomorphism:
A locally homeomorphic relation is a relation $R$ which "locally looks like a homeomorphisms". That is, for any pair of elements $x \in X$ and $y \in Y$, such that $(x, y) \in R$ there exit neighborhoods $U_x$ and $V_y$ and a homeomorphism between them, whose graph coincides with the restriction of the relation $R$.