Let $X$ be a topological space and let $a$ be a point of $X$. There are three interesting topologies on $\pi_1(X,a)$: the quotient topology of the topology of compact convergence, and two other one which you'll let me call admissible and adequate.
In all three topologies, the classes of open normal subgroups coincide; moreover they are simultaneously Hausdorff (or not).
As observed, the topology of compact convergence is not a group topology, although the multiplication is separately continuous.
Say a subgroup $H$ of $\pi_1(X,a)$ is admissible if, any $b\in X$ has a neighborhood $V$ such that forany path class $\gamma\in\pi(X,a,b)$ (fundamental groupoid) and any loop $c$ in $V$ based at $b$, $\gamma c \gamma^{-1}$ belongs to $H$.
There is a unique group topology on $\pi_1(X,a)$ such that the normal admissible subgroups form a basis of neighborhoods of the identity. For this topology, the open subgroups are the admissible subgroups.
If $X$ is locally arcwise connected, then a subgroup is admissible if and only if it is the stabilizer of a point above $a$ in a covering of $X$.
The admissible topology is coarser than the topology of compact convergence, and may be strictly coarser.
- Let us say that a subgroup $H$ of $\pi_1(X,a)$ is adequate if, for any $b\in X$ and any path class $\gamma\in\pi(X,a,b)$, there exists a neighborhood $V$ of $b$ such that for any loop $c$ in $V$ based at $b$ $\gamma c \gamma^{-1}$ belongs to $H$. (Note the switch in the order of quantifiers.)
There exists a unique group topology on $\pi_1(X,a)$ for which adequate subgroups form a basis of open subgroups.
This topology is finer (and possibly strictly finer) than the topology of compact convergence, to which it coincides. However if $X$ is locally arwise connected, both topologies have the same open subgroups.