The question is as given in the title:

Which finite groups are not the automorphism group of some rooted finite tree?

A rephrasing could be: Is any finite group representable as the automorphism group of a finite tree? If not, what is typically unrepresentable?

In case of ambiguity: a homomorphism of finite rooted trees must preserved the root, and so does an isomorphism which is called an automorphism.

**Context**:

The motivation/spirit of the question is as follows. I make the isomorphism classes of finite graphs smaller by specifying a group acting on the graph's vertices, that is an isomorphism must now respect the group action (instead of the bigger $S(n)$ action).
Do I lose something by restricting myself to tree automorphisms instead of considering the group action?