The 2-adic rationals $\mathbb{Q}_2$ and the 3-adic rationals $\mathbb{Q}_3$ are homeomorphic, because each one is a countable disjoint union of Cantor sets.  They are also isomorphic as groups if you assume the axiom of choice, because they are both fields of characteristic 0 and therefore vector spaces over $\mathbb{Q}$ (of the same cardinal dimension).  However, the 2-adic integers $\mathbb{Z}_2$ are a compact subgroup of $\mathbb{Q}_2$ in which every element is infinitely divisible by 3.  On the other hand, in $\mathbb{Q}_3$, any non-trivial sequence $x, x/3, x/9, \ldots$ is unbounded in the complete metric, and is therefore not contained in a compact subgroup.

----------

Keith Conrad asks whether these is an example without the axiom of choice, and Jason De Vito asks whether there is an example using Lie groups.  In fact, there is a cheap example using disconnected Lie groups.  Let $G$ and $H$ be two connected Lie groups that are homeomorphic but not isomorphic.  For instance, abelian $\mathbb{R}^3$, the universal cover $\widetilde{\text{SL}(2,\mathbb{R})}$, and the Heisenberg group of upper unitriangular, real $3 \times 3$ matrices are all homeomorphic, but not isomorphic.  If $G'$ and $H'$ are $G$ and $H$ with the discrete topology, then $G' \times H$ and $G \times H'$ are explicitly isomorphic and explicitly homeomorphic.  But they are not continuously isomorphic, because the connected component of the identity is $G$ for one of them but $H$ for the other one.