I think the answer to the question as literally stated is "the trivial group", but I think there are related inquiries which get into some deep combinatorics.

One way of thinking about the rooted tree operad is that it is the free operad $O(F)$ generated by the Joyal species $F$ (a functor $\mathbb{P} \to Set$ where $\mathbb{P}$ is the groupoid of finite sets $\{1, \ldots, n\}$ and permutations) where $F(0)$ is empty and $F(n)$ is a singleton for $n \geq 1$. You can think of the element of $F(n)$ as a "sprout" $s_n$ consisting of a root, $n$ leaves, and no other nodes, and then the elements of $O(F)$ are obtained recursively by starting with sprouts and applying grafting operations.

So we're looking at operad automorphisms $\phi: O(F) \to O(F)$. By freeness, the endomorphisms of $O(F)$ are in bijection with natural transformations $\psi: F \to U O(F)$ where $U O(F)$ is the underlying species or permutation representation of $O(F)$. Concretely, to give such a natural transformation is to give a collection of trees $t_n = \psi_n(s_n)$ for all $n \geq 1$ where each $t_n$ must be invariant under permuting the leaves, since the sprout $s_n$ is invariant under such permutations. That's a pretty strong condition on $t_n$, and there are actually precious few such collections.

But now you want more: you want $\phi$ to be an automorphism as well. So each sprout $s_n$ must be in the image of $\phi_n$. But no nonsprout tree $u$ can ever map to $s_n$ under $\phi_n$, because if $u$ is obtained by grafting together more than one sprout $s_k$, then $\phi_n(u)$ is obtained by similarly grafting together more than one tree $t_k$, and this is never a sprout.

So in order for there to exist $u$ such that $\phi_n(u) = s_n$, we must have $u = s_n$. To have that for all $n$ means $\psi(s_n) = s_n$ for all $n$, hence the only operad automorphism is the identity automorphism.

I think a more interesting inquiry is to understand the groupoid of rooted trees and isomorphisms between them. This is an incredibly rich object.

Edit: Let me make my last suggestion more precise. Let's define a rooted tree to be a finite set $X$ equipped with a function $f: X \to X$ and an element $r \in X$ such that $f^{(n)}(X) = \{r\}$ for sufficiently large $n$. The idea is that $f(x)$ is one step closer to the root than $x$, unless $x$ is the root. Then an isomorphism is a function $\phi: X \to Y$ which preserves the stepping-closer function and the root. It is determined by its restriction to the leaf set.

But even this groupoid isn't *that* mysterious; it seems automorphism groups are iterated wreath products of symmetric groups. Here is a different but related inquiry which I think is rather more interesting: regarding trees $T$ as posets, describe the category of order-preserving bijections between trees.