I suppose the main point is that the typically studied random graph models are not directed or weighted and they generally don't have self loops. Under your correspondence, this means they are limited to symmetric matrices whose diagonal entries are zero and whose off-diagonal entries are either zero or one - a rather specific class of random matrices, and not really the sort that are of primary interest in random matrix theory. Sometimes these assumptions are relaxed, but the theory is already pretty hard without doing so.
More substantively, the motivation for the two subjects is pretty different. Random graphs were first introduced in combinatorics to prove the existence of interesting graphs by calculating that they occur in random models with positive probability. More recently the field has been influenced by applications to biology, social networks, and computer science. As a result, the problem is usually to understand things like the degree distribution or the expected isoperimetric constant of the graph. Additionally, an important focus of the theory is on developing a given graph according to some random process which potentially adds both nodes and edges (e.g. preferential attachment models) - this limits the possible interactions with random matrix theory, which usually fixes the size of the matrix.
Random matrix theory, on the other hand, was introduced in nuclear physics and taken up more recently by number theorists. In both cases the primary interest is in spectral theory: in physics the idea is that various physically observed quantities appear to have the same distribution as the gaps between eigenvalues of random matrices, and in number theory the distribution of gaps between zeros of the Riemann zeta function also to obey this distribution.
Of course, there are well-known connections between the geometry / combinatorics of graphs and their spectral theory, so it is in principle possible to use these connections as a bridge between random graphs and random matrices. I could be under-informed, but I think the connections are too complicated and a bit too loose to solve hard problems in one area using techniques from the other - particularly because the random models tend to be so different.
