Let $X$ be a topological space and let $a$ be a point of $X$. 
There are three interesting topologies on $\pi_1(X,a)$: the quotient
topology of the topology of compact convergence, and two other one which
you'll let me call admissible and adequate.

In all three topologies, the classes of open normal subgroups coincide;
moreover they are simultaneously Hausdorff (or not).

1) As observed, the topology of compact convergence is not a group topology,
although the multiplication is separately continuous.

2) Say a subgroup $H$ of $\pi_1(X,a)$ is admissible if, any $b\in X$
has a neighborhood $V$ such that forany path class $\gamma\in\pi(X,a,b)$ (fundamental groupoid) and any loop $c$ in $V$ based at $b$, 
$\gamma c \gamma^{-1}$ belongs to $H$.

There is a unique *group topology* on $\pi_1(X,a)$ such that the normal admissible
subgroups form a basis of neighborhoods of the identity. For this topology, the open subgroups are the admissible subgroups.

If $X$ is locally arcwise connected, then a subgroup is admissible if and only if
it is the stabilizer of a point above $a$ in a covering of $X$.

The admissible topology is coarser than the topology of compact convergence,
and may be strictly coarser.

3) Let us say that a subgroup $H$ of $\pi_1(X,a)$ is *adequate* if, for any $b\in X$
and any path class $\gamma\in\pi(X,a,b)$, there exists a neighborhood $V$ of $b$ 
such that for any loop $c$ in $V$ based at $b$
$\gamma c \gamma^{-1}$ belongs to $H$. (Note the switch in the order of quantifiers.)

There exists a unique group topology on $\pi_1(X,a)$ for which adequate subgroups form a basis of open subgroups.

This topology is finer than the topology of compact convergence,
to which it coincides if $X$ is locally arwise connected.