If $G$ is just an ordinary set-theoretic group, then the answer to the question in the title is yes: the automorphisms of $G$ as a (left) $G$-set are all of the form "multiply (on the right) by an element of $G$."
I'm trying to understand the case where $G$ is a group scheme; then as I understand it the stack of $G$-torsors is equivalent to the stack quotient $[\ast/G]$, the stackification of the fibered category whose fiber over a scheme $X$ is a single object with $G(X)$ automorphisms. The image of that single object in the stackification is the trivial $G$-torsor on $X$ (namely the base change $G_X$ of $G$ to $X$) and so the automorphisms of that trivial $G$-torsor should again be $G(X)$.
But consider the case $G=\mu_2$, the group of square roots of unity, which is represented by $\mathrm{Spec}\bigl(\mathbb{Z}[x]/(x^2-1)\bigr)$. Letting $R$ be the ring $\mathbb{F}_2[\varepsilon]/(\varepsilon^2)$, we have $G(R) = \{1,1+\varepsilon\}$. These give us two automorphisms of $G_R = R[x]/(x^2-1)$, namely $x\mapsto x$ and $x\mapsto (1+\varepsilon)x$. But $G_R$ has two more automorphisms that commute with the action by $G(R)$; we can send $x\mapsto x+\varepsilon$ or $x\mapsto (1+\varepsilon)x + \varepsilon$. So how can this be the trivial $G$-torsor over $R$ if it has too many automorphisms as a $G$-object?