Let me add a few further remarks to the above very good answer. I think that the problem of classifying principal bundles is one of the most fundamental questions and applications of algebraic topology.

The basic reason why the classification of principal bundles for $G$ a compact Lie group is so much more complicated than for a finite group is that the compact Lie group is not discrete and has higher homotopy. For example, the principal $G$-bundles over $S^n$ are classified by $\pi_n(BG)\cong \pi_{n-1}(G)$. In particular, the complete classification of bundles is not even known over spheres of arbitrary dimension.

The question simplifies a bit if you stabilize, i.e., you look at $O(\infty)$-bundles or $U(\infty)$-bundles. In these cases, the homotopy sets $[M,BO]$ or $[M,BU]$ are topological K-groups and can in principle be computed using long exact sequences and such things.

However, in general, the classification of principal bundles over finite CW-complexes is going to be more and more complicated with growing dimension. To give you some flavour of the sort of results to expect, you might want to have a look at some of the following papers:

A.Dold and H.Whitney. Classification of oriented sphere bundles over a $4$-complex. Ann. of Math. (2) 69 (1959), 667-677.

I.M.James and E.Thomas. An approach to the enumeration problem for non-stable vector bundles. J. Math. Mech. 14 (1965), 485-506.

F.P.Peterson. Some remarks on Chern classes. Ann. of Math. (2) 69 (1959), 414-420.

L. Smith. Complex 2-plane bundles over $\mathbb{CP}(n)$. Manuscripta Math. 24 (1978), 221-228.

R.M.Switzer. Complex 2-plane bundles over complex projective space. Math. Z. 168 (1979), 275-287.

... and for something more recent (look at the progress in dimension)...

M. Cadek and J. Vanzura. On oriented vector bundles over CW-complexes of dimension 6 and 7. Comment. Math. Univ. Carolin. 33 (1992), 727-736.

B. Antieau and B. Williams. On the classification of oriented 3-plane bundles over a 6-complex. arXiv:1209.2219.

This does not even say anything about the classification of principal bundles with exceptional structure groups....

[Edit:] I should have said that in all the above cases, the results are proved using obstruction theory. The answer then classifies bundles in terms of characteristic classes in suitable cohomology theories, together with additional data like compatibilities with Steenrod operations etc. This is a standard procedure in algebraic topology. Look at Hatcher's algebraic topology book for an introduction to Postnikov towers and obstruction theory.

classifying space$B G$ for G-principal bundles. See ncatlab.org/nlab/show/classifying+space $\endgroup$issimilar to the case of finite gauge groups: if G is discrete then maps $X \to BG$ are the same as maps $\pi_1(X) \to G$ and the notion of homotopy translates. One thing to realize is that also $Hom(\pi(X),G)/G$ is less trivial than it may seem. It's a moduli space of flat connections, which harbours some non-trivial theory, see ncatlab.org/nlab/show/moduli+space+of+connections $\endgroup$8more comments