Speaking as someone whose thesis was also in algebraic graph theory but who has later gone on to do research in other areas, I would say that it is definitely possible to switch fields. The main skills you need are management skills: the ability to manage your own time so that you can spend some time learning a new field while also producing something that others will value, and the ability to manage other people's view of your work. In this regard, I believe that Deane Yang's comment is right on the money. You probably can't afford to drop your initial specialty abruptly and produce no results while you retrain yourself. But if you manage things carefully then anything is possible, regardless of whether you have tenure or are switching to a field that requires a lot of background study.

There are some things you can do to help you solve these management problems. If you can find overlap between your current field and your new field, that will obviously help. Mathematics is so interconnected that this is usually not too hard; in the specific case of algebraic graph theory versus algebraic number theory, the first topic to come to mind is the Ihara zeta function of a graph, but I'm sure there are others.

In my case, I decided it would help to switch from academics to industry/government. I personally found it easier in industry/government to spend x% of my time producing results that pleased my employers and spending the remaining (100-x)% of my time training myself in a new area. Your mileage may vary, of course; in my case, I found that teaching drained me of too much of my energy, but this is not true of everybody.

One last comment I have is that if you take this route, then you will need the ability to maintain a clear sense of your own identity and goals and not be unduly swayed by other people's categorizations. For example, when I switched out of academics, many others regarded me as "leaving mathematics." In fact I was only leaving academics and not leaving mathematics, and it was important for me to ignore other people's view of the matter. As another example, people will want to pigeonhole you as a "something-ist" (and it seems you have been influenced by this point of view, since you use the phrase "being a graph theorist for the rest of my life"); you should resist this pigeonholing, and instead think of yourself just as someone with certain abilities and interests. Thinking of yourself as either a graph theorist or a number theorist is unnecessarily limiting. (Of course you may need to bill yourself as one or the other for the purposes of managing other people's view of you while you're making a transition, but you should not necessarily believe your own propaganda.)