When does Pontryagin duality generalize? Let $T$ be a locally compact abelian (LCA) group. For any other LCA group $G$, let 
$\hom(G,T)$ be the set of continuous homomorphisms $G\to T$. With the compact-open 
topology, $\hom(G,T)$ is certainly a topological group, but is not in general locally 
compact, even if $T$ is compact. In any case there is an obvious homomorphism 
$$
  \alpha_G : G \to \hom(\hom(G,T),T) 
$$
sending $x\in G$ to the functional $\chi\mapsto \chi(x)$. 
I have three questions: 


*
  
*For what LCA groups $T$ is $\hom(G,T)$ locally compact for all locally compact 
      $G$?
  
*For what groups $T$ is $\alpha_G$ always a (topological) isomorphism?
  
*More generally, is there any full subcategory $\mathcal L$ of the category of LCA 
      groups for which $\hom(G,H)$ is in $\mathcal L$ whenever $G$ and $H$ are in 
      $\mathcal L$?


(Edit: I'm most interested in 1 and 2. If there is some $\mathcal L$ satisfying 3 
where the objects in $\mathcal L$ can be characterized by some interesting topological 
criterion, that would be fantastic. My question is not whether there are "ad-hoc" ways of constructing $\mathcal L$.)
 A: Here's an answer for 1: given a LCA group $T$, $\text{Hom}(G,T)$ is LC for every LCA $G$ iff $T$ is a compact Lie group.
Proof: Clearly this is a sufficient condition, because it holds for connected tori, and is obviously stable under taking closed subgroup.
Conversely suppose the condition is satisfied.
Let $\mathbf{Z}^{(\mathbf{N})}$ denote the discrete free abelian group of countable rank. Then $\text{Hom}(\mathbf{Z}^{(\mathbf{N})},T)=T^{\mathbf{N}}$ is locally compact iff $T$ is compact. 
So $T$ is compact. Let now $D$ be the Pontryagin dual of $T$, so that $D$ is discrete. By the condition and Pontryagin duality, $\text{Hom}(D,G)$ is LC for all LCA $G$.
Consider a discrete abelian group $G_0$ (I'll fix it later). For $F\subset D$ finite subset, define $U_F$ as the set of homomorphisms $D\to G_0$ vanishing on $F$. So the $U_F$ form a basis of clopen neighbourhoods of 0 in $\text{Hom}(D,G_0)$. So one of those, say $U_F$ is compact. If $E$ is the subgroup generated by $F$, this shows that $\text{Hom}(D/E,G_0)$ is compact. Now to specify, let us assume from the beginning we picked the group $G_0=H^{(\mathbf{N})}$, where $H=\mathbf{Q}/\mathbf{Z}$. It is a standard verification that if $D/E$ is nonzero, then $\text{Hom}(D/E,H)$ is not trivial, and an easy consequence is that $\text{Hom}(D/E,G_0)$ is not compact, a contradiction. This shows that $D/E=0$. So $D$ is finitely generated, and thus $T$ is a compact abelian Lie group, as required.
A careful look on the proof shows that otherwise, $\text{Hom}(G,T)$ is not locally compact for the specific choice of $G=\hat{\mathbf{Z}}^{\mathbf{N}}\times\mathbf{Z}^{(\mathbf{N})}$, where $\hat{\mathbf{Z}}=\prod_p\mathbf{Z}_p$ and $\mathbf{Z}_p$ are the $p$-adics.
Edit: the answer for 2. is, as mentioned in another post, that only the circle satisfies the condition. (I initially thought it was trivially true for the 2-torus (or even the trivial group!), while it's trivially false.)
Here's a proof. Taking $G=\mathbf{R}/\mathbf{Z}$, the condition implies that $\text{Hom}(\mathbf{R}/\mathbf{Z},T)$ is nonzero; pick $f$; its kernel is finite and the quotient of $\mathbf{R}/\mathbf{Z}$ by the kernel of $f$ is also isomorphic to the circle, so the image is isomorphic to the circle. Now by Pontryagin duality, whenever $\mathbf{R}/\mathbf{Z}$ stands as a closed subgroup in a LCA group, it stands as a direct factor. So write $T=H\times \mathbf{R}/\mathbf{Z}$. Then for every $G$ we get $\text{Hom}(\text{Hom}(G,T),T)=G\times \text{Hom}(\hat{G},H)\times \widehat{\text{Hom}(G,H)}\times\text{Hom}(\text{Hom}(G,H),H)$. Since by assumption the identity of $G$ into the first factor is an isomorphism for every $G$, it follows that all other 3 factors are zero. Taking $G=\mathbf{Z}$, we deduce $\widehat{\text{Hom}(\mathbf{Z},H)}=0$, so $H=\text{Hom}(\mathbf{Z},H)=0$ by Pontryagin duality, hence $T=\mathbf{R}/\mathbf{Z}$.
A: Piggy-backing on the excellent answer by Yves, I think $\mathbb{R}/\mathbb{Z}$ is the only answer for $T$ in question 2. 
Every compact Lie group $T$ is isomorphic to one of the form $F \times (\mathbb{R}/\mathbb{Z})^n$ for a finite abelian group $F$ and finite $n \geq 0$. A necessary condition for satisfaction of 2. is that the canonical map $\mathbb{Z} \to \hom(T, T)$ is an isomorphism. 
We then have 
$$\hom(T, T) = \hom(T, F \times (\mathbb{R}/\mathbb{Z})^n) \cong \hom(T, \mathbb{R}/\mathbb{Z})^n \times \hom(T, F)$$ 
which contains $\hom((\mathbb{R}/\mathbb{Z})^n, \mathbb{R}/\mathbb{Z})^n \cong \mathbb{Z}^{n^2}$ as an infinite summand (with a complementary summand that is finite). Thus $n = 1$. Next, if $F$ is non-trivial, we have a product decomposition 
$$\hom(T, T) = \hom(\mathbb{R}/\mathbb{Z}, \mathbb{R}/\mathbb{Z}) \times \hom(\mathbb{R}/\mathbb{Z}, F) \times \hom(F, \mathbb{R}/\mathbb{Z}) \times \hom(F, F)$$ 
for which the third and fourth factors are non-trivial, and hence $\mathbb{Z}$ cannot project onto the displayed product. Thus, $F = 0$, which completes the proof. 
Edit: I might as well promote my earlier comment as an answer to question 3, as efficiently reformulated by Yves Cornulier: a suitable full subcategory of LCA-groups that is closed under homs is the category of compactly generated abelian Lie groups. (I make no claim is that this is maximal among subcategories that are hom-closed, however.) 
Edit 2: Actually, looking back on an old $n$-Category Café discussion on related matters, John Baez mentions a paper by Michael Barr which in some sense blows my answer to 3 out of the water, and which I'm sure Daniel Miller would be interested in. Here is the abstract: 

Abstract. Let $\mathcal{G}$ denote the full subcategory of topological abelian groups consisting of the groups that can be embedded algebraically and topologically into a product of locally compact abelian groups. We show that there is a full coreflective subcategory $\mathcal{S}$ of $\mathcal{G}$ that contains all locally compact groups and is $\ast$-autonomous. This means that for all $G$, $H$ in $\mathcal{S}$ there is an “internal hom” $G \multimap H$ whose underlying abelian group is $\hom(G, H)$ and that that makes $\mathcal{S}$ into a closed category with a tensor product whose underlying abelian group is a quotient of the algebraic tensor product. Moreover a perfect duality results if we let $T$ denote the circle group and define $G^\ast = G \multimap T$. This is essentially a new exposition of work originally done jointly with H. Kleisli [Theory Appl. Categories, 8, 54–62].

A: I refer to the  paper available here 
R. Brown, P.J. Higgins, S.A. Morris, ``Countable products of
lines  and circles: their closed subgroups, quotients and duality
properties'',  Math. Proc. Camb. Phil. Soc. 78 (1975) 19-32.
In  this  paper,  we  study  the  class  $\mathcal D_{\Pi}$    consisting  of  all  Hausdorff  Abelian  groups
topologically  isomorphic to  a product  of a  compact  group with  a  countable  product
of copies of  $\mathbb R$ and $\mathbb Z$. In §4, we prove:
THEOREM B. Closed subgroups and Hausdorff quotients of groups in  $\mathcal D_{\Pi}$   are again in
$\mathcal D_{\Pi}$.
This paper also introduces the notion of strong duality for two topological abelian groups $A,B$, namely that not only are they dual in the usual sense, but this duality is inherited by closed subgroups and Hausdorff quotients. The theorem is that strong duality is inherited by closed subgroups and Hausdorff quotients. 
Later: I should point out that the purpose of this answer is to show that something can be said about generalising duality outside the category of locally compact abelian groups. This builds on earlier work of A. Kaplan. 
A: I gave an answer to the second question already on math.stackexchange at https://math.stackexchange.com/questions/124379/why-unitary-characters-for-the-dual-group-in-pontryagin-duality-if-g-is-not-co, but I'll put it here too: if $\alpha_G$ is an isomorphism of topological groups for all locally compact abelian groups $G$ then $T$ must be the unit circle. This explains the importance of the unit circle as a target group. It is discussed in Pontryagin's book on topological groups (see Example 72) and the first volume on abstract harmonic analysis by Hewitt and Ross (p. 424). 
