By some coincidence, I have a student going through this stuff now, and we got to this point this just yesterday.
The definition of $f^{-1}$ is certainly disconcerting at first, but it's not that bad. You'd like to say $$f^{-1}\mathcal{F}(U) = \mathcal{F}(f(U))$$ except it doesn't make sense as it stands, unless $f(U)$ is open. So we approximate by open sets from above. A section on the left is a germ of a section of $\mathcal{F}$ defined in some open neighbourhood of $f(U)$, where by germ I mean the equivalence class where you identify two sections if they agree on a smaller neigbourhood. Even if you're still unhappy with this, the adjointness property tells you that it is the right thing to look at.
Also, some of us work with non-quasicoherent sheaves (e.g. locally constant sheaves or constructible sheaves), so it's nice to have a general construction.
Addendum: In my answer yesterday, I had somehow forgotten to mention the etale space or sheaf as a bunch of stalks $$\coprod_y \mathcal{F}_y\to Y$$ viewpoint discussed by Emerton and Martin Brandenburg. Had you started with this "bundle picture", we would be having this discussion in reverse, because pullback is the natural operation here and pushforward is the thing that seems strange.

