Two points:

1. As you allude to, there are multiple categories of graphs to be interested in. I've had a go at naming some of them [here](https://mathoverflow.net/questions/321897/is-mathrmgraph-cartesian-closed/), including directed/undirected, multi/simple, and various conditions on loops, trying to view them from a common framework as being certain categories of presheaves.

2. Let $Gph$ be the category of directed multigraphs (as usual for category theorists), and let $U: Cat \to Gph$ be the forgetful functor. As you point out, [$U$ has a left adjoint](https://ncatlab.org/nlab/show/free+category) $F: Gph \to Cat$, which sends a graph $\Gamma$ to the category of "paths" in $\Gamma$. This adjunction is even monadic, exhibiting $Cat$ as a category of algebras over $Gph$. In this way, it's reasonable to regard a category as a graph with extra structure. I'd hazard a guess that dually **the functor $F: Gph \to Cat$ is maybe comonadic?** (EDIT: Yes, I'm pretty sure that $F$ preserves all equalizers. It is clearly a conservative left adjoint between complete categories, so it's comonadic by the dual of the [crude monadicity theorem](https://ncatlab.org/nlab/show/monadicity+theorem#the_crude_monadicity_theorem).) In this way, it should be reasonable to regard a graph as a category with extra structure associated to being a free category. This would give some more justification for your approach of viewing a graph as a category with extra structure.