Yes (to the question in the title; no to the question in the first line). You can find an example in this paper by Oguiso and Truong. The variety ``$X$'' should do what you want.
Briefly, let $\omega = (1+\sqrt{3}i)/2$ and let $E$ be the elliptic curve $\mathbb C / (\mathbb Z + \omega \mathbb Z)$. Then $E$ has an automorphism $\tau$ of order $3$ given by multiplication by $\omega$. Let $X$ be the crepant resolution of $(E \times E \times E)/\tau$ with the diagonal action. Then $X$ is a Calabi-Yau threefold, with many automorphisms, induced by the action of $\textrm{SL}_3(\mathbb Z[\omega])$ on $E \times E \times E$. This $X$ is apparently rigid; there are refs in the paper.