The 2-adic rationals $\mathbb{Q}_2$ and the 3-adic rationals $\mathbb{Q}_3$ are homeomorphic, because each one is a countable disjoint union of Cantor sets. They are also isomorphic as groups if you assume the axiom of choice, because they are both fields of characteristic 0 and therefore vector spaces over $\mathbb{Q}$ (of the same cardinal dimension). However, the 2-adic integers $\mathbb{Z}_2$ are a compact subgroup of $\mathbb{Q}_2$ in which every element is infinitely divisible by 3. On the other hand, in $\mathbb{Q}_3$, any non-trivial sequence $x, x/3, x/9, \ldots$ is unbounded in the complete metric, and is therefore not contained in a compact subgroup.
Keith Conrad asks whether these is an example without the axiom of choice, and Jason De Vito asks whether there is an example using Lie groups. In fact, there is a cheap example using disconnected Lie groups. Let $G$ and $H$ be two connected Lie groups that are homeomorphic but not isomorphic. For instance, abelian $\mathbb{R}^3$, the universal cover $\widetilde{\text{SL}(2,\mathbb{R})}$, and the Heisenberg group of upper unitriangular, real $3 \times 3$ matrices are all homeomorphic, but not isomorphic. If $G'$ and $H'$ are $G$ and $H$ with the discrete topology, then $G' \times H$ and $G \times H'$ are explicitly isomorphic and explicitly homeomorphic. But they are not continuously isomorphic, because the connected component of the identity is $G$ for one of them but $H$ for the other one.