If a finite $p$-group $P$ acts on a finite-dimensional $CW$-complex $X$ which is acyclic mod $p$, then the fixed point set $X^P$ is also acyclic mod $p$. This is a special case of "Smith theory" (see Theorem II of "[Fixed-Point Theorems for Periodic Transformations][1]", P. A. Smith, American Journal of Mathematics Vol. 63, No. 1 (Jan., 1941), pp. 1-8 for an early version of the theory where $P$ is cyclic).

For a cellular action of $P=\mathbb{Z}/p$ on a contractible finite $CW$ complex with only finitely many fixed points, the reduced mod $p$ cellular chain complex would be a bounded acyclic chain complex of $\mathbb{F}_pP$-modules, with all components free modules in non-zero degree, and the degree zero component the direct sum of a free module and a trivial module of dimension one less than the number of free points. But this is impossible by representation theory if that trivial module is non-zero, since a trivial $\mathbb{F}_pP$-module doesn't have finite projective dimension.

  [1]: http://www.jstor.org/stable/2371271