Two points:

1. As you allude to, there are multiple categories of graphs to be interested in. I've had a go at naming some of them [here](https://mathoverflow.net/questions/321897/is-mathrmgraph-cartesian-closed/), including directed/undirected, multi/simple, and various conditions on loops, trying to view them from a common framework as being certain categories of presheaves.

2. Let $Gph$ be the category of directed multigraphs (as usual for category theorists), and let $U: Cat \to Gph$ be the forgetful functor. As you point out, $U$ has a left adjoint $F: Gph \to Cat$, and is even monadic, exhibiting $Cat$ as a category of algebras over $Gph$. In this way, it's reasonable to regard a category as a graph with extra structure. I'd hazard a guess that dually **the functor $F: Gph \to Cat$ is maybe comonadic?** In this way, it should be reasonable to regard a graph as a category with extra structure associated to being a free category. This would give some more justification for your approach of viewing a category as a graph with extra structure.