If the variety $X$ is defined over $\mathbb C$, is compact and smooth, then it should be possible to make it rigid. Indeed, the variety $Aut(X)$ has at most countable number of irreducible components and all these components should be of bounded dimension $N$.


Let us prove it will be sufficient to fix $N+1$ points (where was $N$ here before the comment of Brian). For every component Comp of $Aut(X)$ let us consider the subvariety of $X$ that is moved by at least one element of this component Comp. It is clear that moved point will be an open subset of $X$, so all of them will have an intersection (a countable intersection of open subsets is non-empty). So there will be a point on $X$ that is moved by at least on element from every component of $Aut(X)$. Let us mark this point, call it $P$. Note now that the components of $Aut(X,P)$ have dimensions at most $N-1$. So we proceed by induction.

In fact we did not really used smoothness of $X$ but we surely used that $X$ is defined over $\mathbb C$.