That is because surfaces of higher genus have a non-zero simplicial volume. It is a general fact that all selfmaps of closed manifolds with non-zero simplicial volume have degree either $-1,0$ or $+1$. (see for example [here][1])

I do not know about the other cases which are mentioned in the footnote.

**EDIT:** As Bruno Martelli pointed out in a comment, Gromov conjectured that closed aspherical manifolds of non-zero Euler characteristic have non-zero simplicial volume. Hence, the same argument is likely to apply in many cases which are interesting.

  [1]: http://www.map.him.uni-bonn.de/index.php/Simplicial_volume