To quote Richard Sharpe, Differential Geometry: Cartan's Generalization of Klein's Erlangen Program:
My innocent question, left over from my undergraduate days, was ``Why is differential geometry the study of a connection on a principal bundle?''
and
It became clear to me that Cartan had a subtle and really wonderful idea, which gives a fully satisfying explanation for the study of an Ehresmann connection on a principal bundle.
To find out this explanation, unfortunately, you will need to read Sharpe's book. But Chern, in the forward, gives his answer:
The answer is of course very simple: because Euclidean geometry studies a connection on a principal bundle, and all geometries are in a sense generalizations of Euclidean geometry.
The bundle in Euclidean geometry is the frame bundle of Euclidean space, i.e. the bundle of choices of point and orthonormal basis of tangent vectors at that point, which is a principal bundle over Euclidean space, with gauge group the group of rigid motions fixing a given point, i.e. the orthogonal group. The rigid motions of Euclidean space act in the obvious way on that bundle, becoming precisely the automorphisms of a unique connection on that bundle, the Levi-Civita connection. That connection gives as its parallel transport the obvious translations.
Similarly, taking the group of transformations of affine space which take lines to lines, the affine transformations, we replace that bundle by the bundle of all choices of point and basis of tangent vectors at that point, a principal bundle over affine space. The affine transformations act in the obvious way on that bundle, becoming precisely the automorphisms of a unique connection on that bundle, again the Levi-Civita connection, with the same parallel transport.
We can repeat this idea with other homogeneous spaces, and it succeeds in defining a connection as long as the stabilizer group of a point is reductive. Starting from various different geometries in the sense of Klein's Erlangen programme, we arrive at a description of those geometries as symmetries of a connection. But then we can bend those geometries, by considering all possible connections with given stabilizer group and given representation of that stabilizer group on the Lie algebra of the symmetry group, giving rise to a natural approach to bend Euclidean space into Riemannian geometry, Minkowski space into Lorentzian geometry, and many other examples. You would really need to read Sharpe's book to get a clearer idea of the possibilities.