One rationale for the terminology pullback is, what your "extra structure" over $Y$ is a vector bundle, of more generally, fiber bundle, $V \to Y$, then the total space of $f^*V$ together with its projection, sit in a pullback square (in the sense of category) with $X \to Y$ on the bottom and $f^*V \to V$ on the top. If your "extra stucture" cannot be thought of as a having an underlying object and a map down to $X$, then you most appeal to what David Robert says- f^*V arsises from a so-called Cartesian-lift.If you're interested in the category theory behind this, look up Grothendeick fibrations. The idea is, if the category of such objects over $X$, say $C_X$ (so for example $C_X$=vector bundles over $X$) depend contravariantly on $X$, then one can pullback objects of $C_Y$ to those in $C_X$ by using a Cartesian lift. If it instead, the dependence is covariant, you can an opcartesian lift to pushforward objects of $C_X$ to $C_Y$. If the dependence goes both ways, then we can do both. If you really want to get a hang on this, try working this out for some examples you know and see that it "spits out" what you expect.
It's worth noting, that the use of lifts is not strictly necessary, depending on how you are given your data. Essentially, there are two different ways of looking at (pseudo)functors from a category into the category of categories (e.g. $VectBun:X \mapsto VectBun(X)$)- one is as actual functors, and one is as fibered categories. The first way makes is clear already what your induced maps are, whereas, for fibrations, you need to use lifts- but here the lifts resemble taking the pullback in the case of vector bundles, so, it is not a bd way of thinking about it.
