Let me attempt to summarise some basic facts.

Let $G$ be a Lie (say) group, and let's let $M$ and $N$ be smooth compact manifolds as in the question. A *principal $G$-bundle* on $N$ is a bundle with structure group $G$ and fibre $G$, and transition functions given by left translation. Alternatively, a principal $G$-bundle on $N$ is a free $G$-space $\tilde{N}$ with orbit space $N$. Principal $G$ bundles on $N$ have a nice homotopy classification. Namely, the set of isomorphism classes of principal $G$-bundles is in one-to-one correspondence with the set of homotopy classes $[N,BG]$, where $BG$ is a *classifying space* of the group $G$.

Not only that, the classification is *functorial*. Given a principal $G$-bundle $\xi$ on $N$ and a map $f\colon M\to N$, the pullback construction gives you a bundle $f^{\ast}\xi$ on $M$, and the corresponding operation on homotopy classes $f^{\ast}\colon [N,BG]\to [M,BG]$ is simply pre-composition with the homotopy class of $f$. This implies that a homotopy equivalence $f\colon M\to N$ gives a one-to-one correspondence between isomorphism classes of principal $G$-bundles over $N$ and $M$.

Now let $G$ be a subgroup of $Gl(n,\mathbb{R})$. It turns out that rank $n$ vector bundles with structure group $G$ are pretty closely related to principal $G$-bundles. In fact the isomorphism classes of these two objects correspond. You can read about this in these notes (see Proposition 4.1) or in the classic book reference "Fibre bundles" by Dale Husemoller.

Two special cases relevant to your second paragraph:

- If $G=O(1)$ then we are classifying real line bundles. Since $O(1)\cong \mathbb{Z}_2$ we have $BO(1)=K(\mathbb{Z}_2,1)=\mathbb{R}P^{\infty}$, and real line bundles on $M$ are classified by elements of $[M,K(\mathbb{Z}_2,1)]\cong H^1(M;\mathbb{Z}_2)$.

Since you assume $M$ is compact, this will imply that there are finitely many real line bundles on $M$ (up to isomorphism).

- If $G=U(1)$ then $BU(1)=K(\mathbb{Z},2)=\mathbb{C}P^{\infty}$ and complex line bundles are classified by $[M,K(\mathbb{Z},2)]\cong H^2(M;\mathbb{Z})$.

Thus for any compact manifold $M$ with nonzero second Betti number, there are infinitely many non-isomorphic complex line bundles over $M$.

A similar homotopy classification holds for bundles with more general fibres. In particular, given a $G$-space $F$ and a principal $G$-bundle, one can build a fibre bundle with fibre $F$ and structure group $G$. Also you can go the other way; each fibre bundle has an underlying principal bundle (see Husemoller chapter 4). So the answer to your final question is yes.