Let $K$ be a connected compact Lie group. The moduli space of flat $K$-bundles over an $n$-torus is homeomorphic to the character variety $Hom(\mathbb{Z}^n,K)/K$.
The identity component of this space is homeomorphic to $T^n/W$ where $T$ is a maximal torus in $K$ and $W$ is the Weyl group (generalizing the example you gave when $K=SU(m)$).
Here are some examples for $K=SU(2)$:
$Hom(\mathbb{Z},K)/K=[-2,2]$
$Hom(\mathbb{Z}^2,K)/K=$
$Hom(\mathbb{Z}^3,K)/K$ is a 3-dimensional orbifold. Here is video of a continuous family of slices of it.
In general, the moduli space is connected iff $n=1$, or $n=2$ and $K$ is simply connected, or $n\geq 3$ and $K$ is isomorphic to a product of $SU(m)$'s and/or $Sp(k)$'s. And the identity component of the moduli space is simply connected iff $K$ is semisimple (there may be other components that are not simply connected however). Example 2. above shows that $\pi_2$ may not be trivial even if $K=SU(2)$.
Replacing $K$ with a complex (or real) reductive group $G$, we have a similar, although more complicated story.
Please see my paper: Topology of character varieties of Abelian groups (co-authored with C. Florentino) for details about the reductive case (and references for the above mentioned facts).
Also, for the fundamental group of these moduli spaces please read my paper: Fundamental Groups of Character Varieties: Surfaces and Tori (co-authored with I. Biswas and D. Ramras).