Like many math terms, the words "pushforward" and "pullback" do not necessarily have unique rigorous universal definitions. Or at least I don't know if they do. Their informal definitions are exactly as you describe. But in each particular setting, you have to figure out whether they have a proper definition or not.
I will just give some examples (despite you wanting more than that). First, if you have two vector spaces $X$ and $Y$ and a linear map $f: X \rightarrow Y$, then it is reasonable (but not common) to call $f(x)$ the "pushforward" of $x \in X$. Moreover, $f$ induces naturally the adjoint map $ f^* : Y^* \rightarrow X^* $, and it is natural to call $ f^* (\eta) $ the "pullback" of $\eta \in Y^*$. I would not call $f^{-1}(y)$ the "pullback" of $y \in Y$, because it is a set rather than a vector. The idea, I think, is that pushforward and pullback should be functorial in some sense so that they should map an object (here a vector) to another object of the same type (and not a set of objects) but in the other space named in the map.
This generalizes naturally to smooth vector bundles. If you have a vector bundle $X$ over a manifold $M$, another vector bundle $Y$ over $N$, and a bundle map $f: X \rightarrow Y$, then all of the above generalizes naturally to elements of the bundle.
You can also define the pullback of a bundle itself. In other words, instead of viewing elements of a vector bundle as the objects, view the vector bundles themselves as objects. Given a map $f: M \rightarrow N$ and a vector bundle $Y$ over $N$, then there is a natural notion of the pullback of $Y$ as a vector bundle $f^*Y$ over $M$. But there is no notion of a pushforward, because if $f$ is not a diffeomorphism, you won't have the necessary uniqueness and smoothness to define the pushforward as a vector bundle. Of course, if $f$ is a diffeomorphism, you can cheat and define the pushforward as the pullback of the inverse map.
Similarly, given a section $s$ of the bundle $Y$, you can pull it back via the map $f$ to define a (smooth) section $f^*s = s\circ f$ of $f^*Y$. But in the smooth category there is way to pushforward a smooth section.
Everything changes when you switch from bundles to sheaves and from smooth to holomorphic or algebraic objects, because singularities become much more manageable. So pushforward becomes well-defined where they were not in the smooth category. But since I'm not a working expert in this stuff, I'd prefer to leave the details to someone else.