A standard reference for this is Hirsch's Differential Topology textbook. If $X$ is compact near all the topologies you'd like to consider are essentially the same. Sometimes they're called the Whitney topologies, or the $C^\infty$-topology, there are weak and strong variants that are only relevant when $X$ is non-compact.
These spaces have the homotopy-type of the space of continuous maps. The basic idea is to consider $Y$ as a submanifold of some Euclidean space, any continuous map $f : X \to Y$ you can apply a smoothing operator to, then project via the tubular neighbourhood theorem to get a smooth map $X \to Y$ approximating $f$ (in the $C^k$ sense for any $k$ as large as makes sense for your given map $f$), similarly you can apply this construction to families of functions.
Hirsch doesn't bother to get into the details of the Frechet manifold structure on these mapping spaces but it's available. Kriegl and Michor's "Convienient setting for global analysis" is a fairly comprehensive (if daunting) reference for this. But there are other references out there that provide a modest amount of details.
We have one of the founders of this subject online -- Richard Palais. Perhaps he will have some comments eventually.
edit: Back to your question. Spaces of continuous maps $C^0(X,Y)$ are a rather traditional thing to study in algebraic topology. It really depends on what kinds of questions you have about these spaces. For example, if $X$ and $Y$ are Eilenberg-Maclane spaces $C^0(X,Y)$ has much to do with plain old cohomology. If your spaces $X$ and $Y$ have nice cell decompositions, you can frequently get at aspects of the homotopy-type of $C^0(X,y)$ via obstruction theory. But in general these spaces are pretty complicated.