> I do this all the time, and no one complains, but I do not feel that it is right...

Perhaps some unease may arise because there are several concepts and issues involved:

1) Pullback whose opposite is pushout, rather than pushforward. This goes under the heading 
direct/inverse limits in (usual) categories. 

2) Initial/final structures (as in Bourbaki). There is a whole philosophy/machinery about such things: concrete categories. See [The Joy of Cats][1]. I guess this strongly relates
to Grothendieck [fibrations and opfibrations][2] mentioned by David Roberts and David Carchedi.

3) Constructions of "pullback" and "pushforward". a) The inverse image of a sheaf is a kind of pullback (in the sense of (1)), and the direct image of a sheaf is given by composition (no pushouts involved!)

b) Pushforward of cohomology classes is a bit more tricky. The algebraic view is that the pullback (i.e. the induced map) $f^*$ is just the homology functor $h(f)$ applied to the map; and pushforward is something fancy involving integration or Poincare duality. Similarly (but vice versa) for homology. 

The geometric view is that pullback in both homology and cohomology is given by the category theoretic pullback ((1) above) whereas pushforward in both homology and cohomology is given by composition (of a very different kind from that for direct image of a sheaf!). Pullback in homology and pushforward in cohomology are defined only for a restricted class of maps (namely those maps that themselves represent cohomology classes). Still, they are just as natural as induced maps, but with respect to a different set of data; so when it comes to composing a pullback with a pushforward (which amounts to a cup or cap-product), transversality has to be applied, which breaks geometric naturality. (With a hint at Steenrod squares. Of course, nothing ever breaks naturality on the algebraic level.) 
Again, there is a whole philosophy/machinery about this, developed in [Buoncristiano-Rourke-Sanderson, A geometric approach to homology theory][3] (start from Chapter 2). An elementary introduction with pictures is in Chapter 1 of Fenn's *Techniques of Geometric Topology*; another elementary (but partial) introduction is in [Kreck's recent book][4].
A short summary is in section 2 [here][5].


  [1]: http://katmat.math.uni-bremen.de/acc/
  [2]: http://nlab.mathforge.org/nlab/show/Grothendieck+fibration
  [3]: http://books.google.com/books?id=fBA8AAAAIAAJ&lpg=PP1&ots=Rr9nixA1cL&dq=Buoncristiano-Rourke-Sanderson%2520A%2520geometric%2520approach%2520to%2520homology%2520theory&pg=PP1#v=onepage&q&f=false
  [4]: http://www.hausdorff-research-institute.uni-bonn.de/files/kreck-DA.pdf
  [5]: http://front.math.ucdavis.edu/0612.5082