<strike>Joel's answer is a special case of the following.</strike> A variation on Joel's answer is the following.  Let G be a transitive permutation group of continuous maps on a finite topological space X with more than 2 elements. Then G together with all constants is never the whole monoid of all continuous self maps of  X. 

Proof. A finite topological space is just a finite preordered set via the specialization ordering/Alexandrov topology. Continuous maps are precisely order preserving maps. 

If the preorder is the universal equivalence relation we have the indiscrete topology and so all maps are continuous. If the preorder is equality then the topology is discrete and so all maps are continuous. Since there are always maps on a three or more element set which are neither constant nor permutations we are done in these two cases. Let's prove only these two cases occur. 

Suppose that $x\leq y$ and $gx=y$ with $g\in G$ by transitivity.  Then $x\leq gx$ and so $x\leq gx\leq g^2x\leq \cdots$. Since G is finite we eventually get $g^n=1$ and so $y=gx\leq x$. Thus any comparable elements are equivalent.  Hence the preorder is an equivalence relation. If there is more than 1 class and also some class is not a singleton, then crushing the non-singleton is continuous but not in the monoid. So the preorder is equality or the universal equivalence relation.  

<b>Added.</b> The above argument shows that if G acting on a finite set X with more than 2 elements is a primitive permutation group (meaning there is no equivalence relation on X preserved by G except equality and the universal relation) then G can only be the homeomorphism group of a topology on $X$ if $G$ is $S_X$ and the topology on X is discrete or indiscrete. Thus any proper submonoid of self-maps on X whose group of units is primitive is not the monoid of all continuous mappings for any topology on X. This provides another family of examples.