Something like this is worked out in papers of Pawel Gajer:
Gajer, Paweł Higher holonomies, geometric loop groups and smooth Deligne cohomology. Advances in geometry, 195–235, Progr. Math., 172, Birkhäuser Boston, Boston, MA, 1999.
(I suggest you also see its Math Review)
The rough idea is that there's a bona-fide topological group model for the loop space $\Omega M$ when $M$ is smooth and compact. Holonomy is then a bona-fide topological group homomorphism $\Omega M \to G$.