The category of topological spaces and continuous functions does not have canonical notion of a relation. Let me elaborate.

There is only one reasonable 1-categorical notion of a relation. If $\mathbb{C}$ is a category, then to define a concept of a relation in $\mathbb{C}$, you have to decide what you mean by a subobject of an object in $\mathbb{C}$.

The most general way to think of a subobject $\phi$ of an object $A$, is to think of $\phi$ as of a logical formula over $A$ (i.e. the "virtual" subobject $A_0$ of $A$ corresponding to formula $\phi$ is given by a generalized set $A_0 = \{a \in A \colon \phi(a)\}$; in the presence of comprehension, such "virtual" subobjects may be materialized in the category, but the point is that we *do not need* to materialize). Therefore, to define subobjects in $\mathbb{C}$, you have to define a logic over $\mathbb{C}$. The concept of logic over a category is encapsulated by the concept of fiberwise posetal fibration over the category.

Let us assume that $p \colon \mathbb{U} \rightarrow \mathbb{C}$ is such a fibration over $\mathbb{C}$. A relation $\phi \colon A \nrightarrow B$ in $\mathbb{C}$ corresponds to an object $\phi$ in the fibre of $p$ over $A \times B$. The only problem that remains to solve, is to find a way to compose two relations in such a way that the composition is associative and has neutral elements (i.e. identities). It is not hard to see that to define the composition in the natural way (i.e. ${a (\psi \circ \phi) c} \Leftrightarrow {\exists_{b \in B} {a \phi b} \wedge {b \psi c}}$), our logic $p$ has to have stable cartesian connectives and stable existential quantifiers. Category-theorists call such $p$ a regular logic fibration over $\mathbb{C}$. Moreover, if you have a regular logic fibration, then you can take its *resolution* and obtain a 2-posetal category of relations $\mathit{Rel}(\mathbb{C})$ together with a canonical embedding:

$$\mathbb{C} \rightarrow \mathit{Rel}(\mathbb{C})$$

which gives an interpretation of a morhpism from $\mathbb{C}$ as a relation in $\mathit{Rel}(\mathbb{C})$.

Now, every (sufficiently complete) category $\mathbb{C}$ has associated one canonical internal logic --- the logic of canonical subobjects: subobjects associated to monomorphisms. For example, the canonical internal logic of $\mathbf{Set}$ gives the usual notion of a relation between sets and induces the usual category of relations. It is a good exercise to show that the canonical internal logic of a category is regular if and only if the category is regular. Because the category of topological spaces and continuous maps is not regular, there is no canonical notion of a relation between topological spaces. There are three ways to overcome this annoying aspect of topological spaces:

* move to a more general category that is regular,
* move to a regular subcategory,
* take a non-canonical logic over the category of topological spaces that is regular.

Which way is the best way depends on your particular applications (I guess this is the reason why your question was closed --- it is completely unclear what you want to achieve). 

BTW, there is a bit more general notion of a relation (and one may encounter it in category theory --- for example in the definition of sheaves over quantales): we can substitute regular logic with regular monoidal logic; i.e. we can weaken cartesian connectives to monoidal connectives and define the composition of $\phi \colon A \nrightarrow B$ with $\psi \colon B \nrightarrow C$ as ${a (\psi \circ \phi) c} \Leftrightarrow {\exists_{b \in B} {a \phi b} \otimes {b \psi c}}$. To be honest, one may imagine even more general notion of a relation, but I do not think it gives us anything useful in our context, so I will refrain from writing about it.