If both bundles were trivial, say $X\times \mathbb{R}^n\rightarrow X$ and $Y\times \mathbb{R}^n\rightarrow Y$ one could just take $X\times Y\times M(m,n,\mathbb{R})$. Different choices of trivializations should give homeomorphic spaces, so this topology seems to be right.
In general the bundles are just locally trivial. However this is enough to write down a basis for the topology.
I don't see, what nice properties this topology might have but it seems to be the canonical choice.
EDIT: I think one can reduce this to classical vector bundle constructions in the following way. Given two vector bundles $p_i:E_i\rightarrow X_i$ (for $i=1,2$). Let $pr_i:X_1\times X_2\rightarrow X_i$ denote the projections. Then the desired bundle is just Hom$(pr_1^*(p_2,E_2,X_2),pr_2^*(p_1,E_1,X_1))$.
