What is the intuition behind using compact open topology for eg. in the case of Pontryagin dual ?

3$\begingroup$ The reason I use the compact open topology is because it lets me view continuous maps between two spaces as a space. So it gives a function object in the category of spaces and that's what we need for it to be a monoidal category. $\endgroup$ – David White May 10 '13 at 18:59

5$\begingroup$ Wouldn't that go for any topology you put on the set of continuous maps between two spaces? $\endgroup$ – user1688 May 10 '13 at 20:14

1$\begingroup$ It has to be functorial and satisfy the homtensor adjunction (sometimes called Currying). I don't think a random topology would do $\endgroup$ – David White May 10 '13 at 20:47

2$\begingroup$ @David: About "that's what we need for it to be a monoidal category"  You probably meant "monoidal closed". $\endgroup$ – Andreas Blass May 10 '13 at 21:51

4$\begingroup$ []:pages.bangor.ac.uk/~mas010/pdffiles/functionspaces.pdf My paper "Function spaces and product topologies" (1964), available [here][1], showed in the Hausdorff case that the compactopen topology allows for a non symmetric monoidal closed category of topological spaces. The non Hausddorff case was treated later by by Booth and Tillotson(Pacific J Math). $\endgroup$ – Ronnie Brown May 11 '13 at 14:13
Given two spaces $X$ and $Y$, how to define the mapping space betweeen them, i.e. what topology should we put on the set of maps between them?
If $X$ is compact and $Y$ a metric space, this is quite easy as one can put a metric on $Map(X,Y)$: For $f,g\in Map(X,Y)$ define their distance just to be the maximum of the distances between $f(x)$ and $g(x)$ as $x$ ranging over the points in $X$.
If $Y$ is no longer metric, we have to find a replacement what it means for two maps to be close. Say, we have again two maps $f,g\in Map(X,Y)$. Let $K\subset X$ be compact and $U\subset Y$ be open such that $f(K)\subset U$. Assume now that $Y$ is Hausdorff (else, this construction might behave badly anyhow). Then $f(K)\subset Y$ is closed, so you would expect that if you move $f(K)$ a little bit, then it stays inside of $U$. So if $g$ is close to $f$, then $g(K)$ should be still inside of $U$. Thus, it is sensible to define an open neighborhood of $f$ to be all maps $g$ such that $g(K) \subset U$. And actually, this agrees with the metric definition when $X$ is compact and $Y$ metric! Furthermore, as long as you have "enough" compact subsets in $X$ (i.e. $X$ is compactly generated) and everything is Hausdorff, then this topology has all the pleasant properties one likes.
I have a weakness for pictures. So here is one
The above shows a function $f: \mathbb R \to \mathbb R$, a compact set $C$, and an open set $U$. The condition $f(C) \subseteq U$ is that the graph of $f$ passes through the shaded part shown in the picture.

7
The compact open topology is essential for getting compact sets in your function space–especially a version of the ArzelàAscoli theorem holds for spaces of continuous functions to uniform spaces equipped with the compact open topology ($C(X,Y)$ with the compact open topology where $X$ is an appropriate topological space and $Y$ is an appropriate uniform space).
In your case (Pontryagin duality) the uniform space $Y$ is just the circle group $\mathbb{T}$ and $X$ is your group $G$. Consider a compact neighbourhood $U$ of the neutral element $e\in G$, an open neighbourhood $I:=\exp((2\pi i\epsilon, 2\pi i\epsilon))\subset\mathbb{T}$ of 1 where $0<\epsilon<1$. Now there is a maximal set of irreducible representations $S\subset \hat{G}\subset C(G,\mathbb{T})$ such that $\forall \pi\in S\ \pi(U)\subset I$. The definition of the compact open topology guarantees that $S$ is open (it is just the definition). But $S$ is also equicontinuous: For $n\in\mathbb{N}$ define $U_0:=U$,$U_{n+1}:=\{x\in U_n\mid x+x\in U_n\}$. Clearly for $\pi\in S$ we have $\pi(U_n)\subset \exp((2\pi i \epsilon/2^n,2\pi i \epsilon/2^n))$. It suffices to prove equicontinuity at the point $e$ (shifting does not change the situation). Now we can apply ArzelàAscoli: $S$ is in fact precompact and we get a compact neighbourhood $\bar{S}$ of the neutral element of $\hat{G}$ thus $\hat{G}$ is locally compact.
There is also a categorical motivation for this topology: The category of topological spaces is not cartesian closed, thus in this category (and many usual related topologies) the adjunction mentioned by David White in the comment above (currying) does not exist—it is impossible to choose the right topology. However, in the category of compactly generated spaces it works: It is cartesian closed and the topology for function spaces is the compact open topology. Again, the compact open topology guarantees that there are enough compact sets such that the topology is actually compactly generated (I guess that this is also a consequence of a version of ArzelàAscoli using even continuity, which does not need a uniformity, instead of equicontinuity, but I am not sure).
Firstly, it is natural to take this topology, because it's convergence is just compact convergence, or, in the case of locally compact Hausdorff spaces, locally uniform convergence. Then secondly, in the most structural proof of the Pontryagin duality using the GelfandNaimark theorem, you identify the Pontryagin dual with the structure space of the C*algebra and the latter carries a natural topology, the Krull topology and it turns out that in the interpretation as function spaces this just is the compactopen topology.