The short answer is we call it the Gauss–Manin connection because that's what Grothendieck called it. The name is attributed to Grothendieck in two early, seminal pieces: namely, Katz's thesis and a subsequent article of his ("On the differentiation of De Rham cohomology classes with respect to parameters", Katz & Oda).
If you can read (some) French, look through Yves Andre's chapter in "Geometric Aspects of Dwork Theory". If not, check through the start of this arXiv paper (through the first couple paragraphs of 1.2).
The slightly less short answer is the one Gerhard Paseman alluded to above; quoting from the aforelinked arXiv paper ("Towards a nonlinear Schwarz’s list", Philip Boalch):
One reason hypergeometric equations are interesting is that they provide the simplest explicit examples of Gauss–Manin connections. Indeed this is one reason Gauss was interested in them: he observed that the periods of a family of elliptic curves satisfy a (Gauss) hypergeometric equation. (The modern interpretation of this is as the explicit form of the natural flat connection on the vector bundle of first cohomologies over the base of the family of elliptic curves, written with respect to the basis given by the holomorphic one forms—and their derivatives—on the fibres.) Nowadays there is still much interest in such linear differential equations “coming from geometry”.
Thus the nonlinear analogue of the Gauss hypergeometric equation should be the explicit form of the simplest nonabelian Gauss–Manin connection (i.e the explicit form of the
natural connection on the bundle of first nonabelian cohomologies of some family of varieties). The simplest interesting case corresponds to taking the universal family of four punctured spheres and taking cohomology with coefficients in $\mathrm{SL}_2(\mathbb{C})$ (one needs a non-trivial family of varieties with nonabelian fundamental groups).