Frucht showed that every finite group is the automorphism group of a finite graph. The paper is here.

The argument basically is that a group is the automorphism group of its (colored) Cayley graph
and that the colors of edge in the Cayley graph can be coded into an uncolored graph that has the same automorphism group.

This argument seems to carry over to the countably infinite case.

Does anybody know a reference for this?

In the uncountable, is it true that every group is the automorphism group of a graph? (Reference?) It seems like one has to code ordinals into rigid graphs in order to code the uncountably many colors of the Cayley graph.