Yes, this is true. A way to prove this is to use the existence of transcendence basis for field extensions, and the fact that $\overline{\mathbf{Q}_p}$ is algebraically closed.
First, assume $\alpha$ is transcendental. Then there exists a transcendence basis $S$ of $\overline{\mathbf{Q}_p}/\mathbf{Q}$ containing $\alpha$. By permuting the elements of $S$, there exists an automorphism $\sigma$ of $\mathbf{Q}(S)$ such that $\sigma(\alpha) \neq \alpha$. Since $\overline{\mathbf{Q}_p}$ is an algebraic closure of $\mathbf{Q}(S)$, you can extend $\sigma$ to an automorphism of $\overline{\mathbf{Q}_p}$, which gives a contradiction.
Finally, if $\alpha \in \overline{\mathbf{Q}}$ but $\alpha \not\in \mathbf{Q}$, then you can take the Galois closure $L$ of $\mathbf{Q}(\alpha)$ and find an automorphism $\sigma$ of $L$ such that $\sigma(\alpha) \neq \alpha$. Then, repeating the above argument, you can extend $\sigma$ to $\overline{\mathbf{Q}_p}$, which gives a contradiction.
Note that the argument works replacing $\overline{\mathbf{Q}_p}$ by any algebraically closed field of characteristic $0$.