There is only one reasonable 1-categorical notion of a relation: a relation from $A$ to $B$ is a "subobject" of the product $A \times B$. IfTherefore, if $\mathbb{C}$ is a category, then to define a concept of a relation in $\mathbb{C}$, you have to decide what you mean by a subobject of an object in $\mathbb{C}$.
The most general way to think of a subobject $\phi$ of an object $A$, is to think of $\phi$ as of a logical formula over $A$ (i.e. the "virtual" subobject $A_0$ of $A$ corresponding to formula $\phi$ is given by a generalized setelements that satisfy the formula $A_0 = \{a \in A \colon \phi(a)\}$$\{a \in A \colon \phi(a)\}$; in the presence of comprehension, such "virtual" subobjects may be materialized in the category, but the point is that we do not need to materialize --- a relation does not have to be representable in the category; it can belong to another world). Therefore, to define subobjects in $\mathbb{C}$, you have to define a logic over $\mathbb{C}$. The concept of logic over a category is encapsulated by the concept of fiberwise posetal fibration over the category.
Let us assume that $p \colon \mathbb{U} \rightarrow \mathbb{C}$ is such a fibration over $\mathbb{C}$. A relation $\phi \colon A \nrightarrow B$ in $\mathbb{C}$ corresponds to an object $\phi$ in the fibre of $p$ over $A \times B$. The only problem that remains to solve, is to find a way to compose two relations in such a way that the composition is associative and has neutral elements (i.e. identities). It is not hard to see that to define the composition in the natural way (i.e. ${a (\psi \circ \phi) c} \Leftrightarrow {\exists_{b \in B} {a \phi b} \wedge {b \psi c}}$), our logic $p$ has to have stable cartesian connectives and stable existential quantifiers. Category-theorists call such $p$ a regular logic fibration over $\mathbb{C}$. Moreover, ifIf you have a regular logic fibration, then you can take its resolution and obtain a 2-posetal category of relations $\mathit{Rel}(\mathbb{C})$$\mathit{Rel}(p)$ together with a canonical embedding:
$$\mathbb{C} \rightarrow \mathit{Rel}(\mathbb{C})$$$$\mathbb{C} \rightarrow \mathit{Rel}(p)$$
which gives an interpretation of a morhpismmorhpisms from $\mathbb{C}$ as a relationrelations in $\mathit{Rel}(\mathbb{C})$$\mathit{Rel}(p)$.
Now, every (sufficiently complete) category $\mathbb{C}$ has associated one canonical internal logic --- the logic of canonical subobjects: (i.e. subobjects associated to monomorphisms $A_0 \rightarrow A$). For example, the canonical internal logic of $\mathbf{Set}$ gives the usual notion of a relation between sets and induces the usual category of relations. It is a good exercise to show that the canonical internal logic of a (finitely complete) category is regular if and only if the category is regular in the usual sense (i.e. it has stable images). Because the category of topological spaces and continuous maps is not regular, there is no canonical notion of a relation between topological spaces. There are three ways to overcome this annoying aspect of topological spaces:
- move to a more general category that is regular,
- move to a regular subcategory,
- take a non-canonical logic that is regular over the category of topological spaces that is regular.