I think focusing on graphs is not a good idea. We focus on functors for very good reasons. Here are a few:

 - Many diagrams which are used in practice are functors between categories, and forgetting that they are compatible with composition could seem artificial in many cases.
 - We want to compute colimits. A very fundamental tool for this is the notion of colimit-cofinal functor: those functors $u:A\to B$ such that, for any functor $f:B\to C$, the colimit of $f$ exists if and only if the colimit of $fu$ exists, in which case both are isomorphic in $C$. Those functors are characterized by connectivity properties on comma categories, the formation of which is sensitive to compositions. Furthermore, (co)limits are special cases of pointwise Kan left Kan extensions, the formation of which cannot be determined by underlying graphs.
 - Category theory is used in homotopical contexts for a long time, and we now have the theory of $\infty$-categories to explain conceptually how this is possible. Many features of ordinary categories (such as the theory of (co)limits and Kan extensions) are robust enough to be promoted to $\infty$-categories. However, only the version with functors can be transposed to higher categories: if you have a functor $f:I\to C$ from a category $I$ to an $\infty$-category $C$, it is not true that the colimit of $f$ in $C$ can be tetermined by the restriction of $f$ on the graph of $I$ only (think of the kind of homology you would get if you would define cellular homology of a CW-complex by stoping short at its $1$-skeleton). In ordinary category theory, this holds because there is no ambiguity about the notion of composition of two maps in $C$, whereas in a higher category it could be you have a space/category of possible compositions of two maps that cannot be ignored. Therefore, if we have in mind possible generalizations of category theory to homotopical or higher categorical context, focusing on underlying graphs is very misleading. 

There are instances where the point of view of graphs is very useful, though. For instance, there is Nori's construction of an abelian category of motives, which relies heavily on singular cohomology seen as a map of graphs; this is documented in this book of [Annette Huber and Stefan Müller-Stach][1], for instance.

If you really want to focus on graphs, then I guess that, in a monograph on category theory, you may write a chapter explaining why we naturally speak the language of graphs when we express ourselves in the lagnuage of category theory. For instance, the category of small category is monadic on the category of graphs. In particular, any category is a colimit of free categories. There are many fundamental exemples of free categories, and is true that, when we write a diagram explicitely, we only write the images of generators, because this is what working on free objects is good for. It is also interesting to see how caterical constructions (such as colimits) are compatible with the presentabions of categories as colimits of free ones. But you will see then that the colimits of interest in this respect are in fact those which are equivalent to their corresponding 2-colimits (i.e. you will start do do homotopy theory where weak equivalences are equivalences of categories). Expressing a given category as $2$-colimits of elementary free categories can be instructive. Cases where we have such a nice inductive procedure of this kind are interesting in practice: this is what is happening with direct Reedy categories, for instance.


  [1]: http://home.mathematik.uni-freiburg.de/arithgeom/preprints/buch-errata/buch-errata.pdf