There's a "really close correspondence" between quivers and categories, where quivers are directed graphs that can have multiple arrows from one vertex to another one and also loop arrows, which are arrows from a vertex to itself. Isomorphisms become undirected edges. This is a really good and precise way to think about it, because this viewpoint generalizes very nicely to some models of higher catgory theory, specifically A. Joyal's theory of quasicategories. The whole beauty of category theory is that all of the information about an object is contained within its arrows, and that the underlying thing that the category represents is not actually important. That is, we have all of the information about the category by: a.) Knowing the structure of the graph of the category. b.) knowing the structure of the hom-sets (which don't always have to be sets), and c.) in extra structure that lives over the graph (like a grothendieck topology or a model structure (this is unrelated to the models you were talking about. It has to do with abstract homotopy theory). The only place that it's nice to have sets is for defining the hom-sets in an unenriched setting. Without some notion of a set, it's hard to get important theorems like yoneda's lemma. Lawvere famously came up with two categorical foundational theories, ETCC and ETCS. At the moment, ETCC is pretty much useless. It contains ETCS as a subaxiomatization, but all of the structure axiomatized in ETCC can be constructed from ETCS (depending on if you take the topos of sets to be boolean or not, and some other unimportant technicalities).
ETCS = Elementary theory of the category of sets
ETCC = Elementary theory of the category of categories