A few points:

1. The category of graphs is certainly not _equivalent_ to the category of categories. But they are related (for more on that see (3)).

2. As you allude to, there are multiple categories of graphs to be interested in. I've had a go at naming some of them [here](https://mathoverflow.net/questions/321897/is-mathrmgraph-cartesian-closed/), including directed/undirected, multi/simple, and various conditions on loops, trying to view them from a common framework as being certain categories of presheaves.

3. Let $Gph$ be the category of directed multigraphs (as usual for category theorists), and let $U: Cat \to Gph$ be the forgetful functor. As you point out, [$U$ has a left adjoint](https://ncatlab.org/nlab/show/free+category) $F: Gph \to Cat$, which sends a graph $\Gamma$ to the category of "paths" in $\Gamma$. This adjunction is even monadic, exhibiting $Cat$ as a category of algebras over $Gph$. In this way, it's reasonable to regard a category as a graph with extra structure. I'd hazard a guess that dually **the functor $F: Gph \to Cat$ is maybe comonadic?** (EDIT: Yes, I'm pretty sure that $F$ preserves all equalizers. It is clearly a conservative left adjoint between complete categories, so it's comonadic by the dual of the [crude monadicity theorem](https://ncatlab.org/nlab/show/monadicity+theorem#the_crude_monadicity_theorem).) In this way, it should be reasonable to regard a graph as a category with extra structure associated to being a free category. This would give some more justification for your approach of viewing a graph as a category with extra structure.