The question is as given in the title:
Which finite groups are not the automorphism group of some rooted finite tree?
A rephrasing could be: Is any finite group representable as the automorphism group of a finite tree? If not, what is typically unrepresentable?
In case of ambiguity: a homomorphism of finite rooted trees must preserved the root, and so does an isomorphism which is called an automorphism.
The motivation/spirit of the question is as follows. I make the isomorphism classes of finite graphs smaller by specifying a group acting on the graph's vertices, that is an isomorphism must now respect the group action (instead of the bigger $S(n)$ action). Do I lose something by restricting myself to tree automorphisms instead of considering the group action?