So in the standard model of particle physics, there exist particles with fractional charge. What this means geometrically is as follows: We are given a smooth manifold with a principal $U(1)$ bundle $P$. Then, as far as I understand, we can construct the associated vector bundle to the one-dimensional representation $\phi \mapsto e^{i\theta/2}\phi$, where $\theta \in \mathbb{R}$. (For a charge of 1/2, or more generally $\phi \mapsto e^{iq\theta}\phi$).
Now, this isn't actually a representation of $U(1)$, it's a representation of a double-cover $U(1) \rightarrow U(1)$, $z \mapsto z^2$ (a somewhat similar problem arises in constructing bundles associated to spin representations). So to proceed with the geometric construction one needs to construct a double-cover of the bundle $U(1)$.
However, particles with fractional charge generally arise in theories with $SU(N)$ also acting. Explicitly, one situation is where we have a manifold $M$ with a principal $U(1) \times SU(2)$ bundle $P$, and we want to construct an associated vector bundle to the representation $\frac{1}{2} \otimes V$, where $V$ is the defining 2-dimensional complex representation of $SU(2)$ and $\frac{1}{2}$ denotes the 'half-charge representation' of $U(1)$. So we need to construct a bundle corresponding to the double cover $U(1) \times SU(2) \rightarrow U(1) \times SU(2)$ which is the identity on the second factor.
A similar thing occurs with quarks, which transform under an $SU(3)$ symmetry, and also have charges with denominators 3.
My question is: Is there a geometric reason why the factor $SU(2)$ or $SU(3)$ allows for fractional charge? Does having a principal $U(1) \times SU(N)$ bundle canonically give an $n-$fold cover? Page 16 of this paper: http://arxiv.org/abs/hep-th/9701069 seems to suggest it's related to the fact that $SU(N)$ has a center consisting of $N$ elements (more specifically, isomorphic to $\mathbb{Z}/n\mathbb{Z}$), but I can't quite follow the argument.
Thanks, and also please correct me if my understanding of the situation is off.