That is because surfaces of higher genus have a non-zero simplicial volume. It is a general fact that all selfmaps of closed manifolds with non-zero simplicial volume have degree either $-1,0$ or $+1$. (see for example here)
I do not know about the other cases which are mentioned in the footnote.
EDIT: As Bruno Martelli pointed out in a comment, Gromov conjectured that closed aspherical manifolds of non-zero Euler characteristic have non-zero simplicial volume. Hence, the same argument is likely to apply in many cases which are interesting.