What sets of self-maps are the continuous self-maps under some topology? An open question on MSE, https://math.stackexchange.com/questions/427634/a-topology-such-that-the-continuous-functions-are-exactly-the-polynomials , asks whether there is an infinite field and a topology over that field such that the continuous functions from the field to itself are precisely the polynomials (and whether such a topology exists for $\Bbb Q$, $\Bbb R$, or $\Bbb C$). Another question, linked from there, asks the same about holomorphic functions. The common notion is this:
Given a set $X$ and a set $\mathscr C$ of functions from $X$ to $X$, what are necessary and/or sufficient conditions under which there is a topology $\mathscr T$ on $X$ such that a function $f\colon X\to X$ is $\mathscr T$-$\mathscr T$ continuous iff $f\in \mathscr C$.
It is immediately clear that $\mathscr C$ must contain the identity function and all constant functions, and must be closed under composition. Nothing else seems terribly obvious. What is known about this?
Edit: for the purpose of answering the specific questions on MSE, even results limited to connected, bihomogeneous $T_1$ spaces would be very helpful.
 A: Let me point out that the properties stated at the end are not
sufficient.
Observation. There is a set $X$ with a set $F$ of functions
on $X$ that contains the identity function and all constant
functions and is closed under composition, but is not the set of
continuous functions with respect to any topology on $X$.
Proof. Let $X=\{1, 2, 3, 4\}$, and let $F$ have the identity
function, the constant functions and the permutations $(1 2)$, $(3
4)$ and $(1 2)(3 4)$. This class of functions has the identity and
constant functions and is closed under composition. But I claim it
is not the set of continuous functions on $X$ for any topology.
Suppose towards contradiction that $\tau$ is a topology on $X$
with $F$ being the set of all continuous functions. Note that the
permutations in $F$ are self-inverse, and so they are actually
homeomorphisms with respect to $\tau$. It follows that if either
$1$ or $2$ is isolated, then so is the other, and the
same with $3$ and $4$. Because the function
$({{1234}\atop{1134}})$ is not in $F$, it follows that $\tau$ must
have an open set $U$ with $1\in U$ and $2\notin U$. Similarly,
there is an open set $V$ with $3\in V$ and $4\notin V$.
Let us now consider various cases. If either $U$ or $V$ is a
doubleton, like $\{1,3\}$, then it easily follows that $\{1,4\}$,
$\{2,3\}$ and $\{2,4\}$ are also open, and so $\tau$ is discrete,
contrary to the fact that $F$ does not contain all functions on
$X$. If both $U$ are tripletons, then $U\cap V=\{1,3\}$ is a
doubleton, and we are in the previous case. So one of them must be
a singleton. Assume without loss that $U=\{1\}$ is a singleton,
and so both $1$ and $2$ are isolated. Since $\tau$ is not
discrete, it follows that $3$ and $4$ are not isolated, and so we
know that $\{1,2,3\}$ and $\{1,2,4\}$ are open and any open set
containing either $3$ or $4$ contains both $1$ and $2$. In this
case, the function $({{1234}\atop{1233}})$ would be continuous,
but it is not in $F$. So there is no such topology $\tau$. QED
Similar examples can be constructed from disjoint unions of
spaces, where one doesn't allow all piecewise constant functions into
$F$.
A: Joel's answer is a special case of the following. A variation on Joel's answer is the following.  Let G be a transitive permutation group of continuous maps on a finite topological space X with more than 2 elements. Then G together with all constants is never the whole monoid of all continuous self maps of  X. 
Proof. A finite topological space is just a finite preordered set via the specialization ordering/Alexandrov topology. Continuous maps are precisely order preserving maps. 
If the preorder is the universal equivalence relation we have the indiscrete topology and so all maps are continuous. If the preorder is equality then the topology is discrete and so all maps are continuous. Since there are always maps on a three or more element set which are neither constant nor permutations we are done in these two cases. Let's prove only these two cases occur. 
Suppose that $x\leq y$ and $gx=y$ with $g\in G$ by transitivity.  Then $x\leq gx$ and so $x\leq gx\leq g^2x\leq \cdots$. Since G is finite we eventually get $g^n=1$ and so $y=gx\leq x$. Thus any comparable elements are equivalent.  Hence the preorder is an equivalence relation. If there is more than 1 class and also some class is not a singleton, then crushing the non-singleton is continuous but not in the monoid. So the preorder is equality or the universal equivalence relation.  
Added. The above argument shows that if G acting on a finite set X with more than 2 elements is a primitive permutation group (meaning there is no equivalence relation on X preserved by G except equality and the universal relation) then G can only be the homeomorphism group of a topology on $X$ if $G$ is $S_X$ and the topology on X is discrete or indiscrete. Thus any proper submonoid of self-maps on X whose group of units is primitive is not the monoid of all continuous mappings for any topology on X. This provides another family of examples. 
A: Here is another example, but with a somewhat different flavor than the others posted here.  Namely, let $\mathbb{Q}$ be the set of rational numbers, and consider the collection of all functions from $\mathbb{Q}$ to $\mathbb{Q}$ which can be extended to continuous functions from $\mathbb{R}$ to $\mathbb{R}$.  (Here $\mathbb{R}$ is the set of real numbers, endowed with the Euclidean topology.)  The article "Can a subset's topology detect continuous extensions?" (The College Mathematics Journal, Volume 49, 2018 - Issue 2) contains a proof that there is no topology $\mathcal{T}$ for which this collection equals the collection of all continuous self-maps of $\mathbb{Q}$ with respect to $\mathcal{T}$.
A: Added.Let me explain better how this answer relates to the question since the OP asked for clarification. Being the monoid of all continuous maps is an algebraic property in the following sense. M is a monoid of mappings on a set X containing all constant maps iff M has a minimal ideal consisting of left zeroes and M acts faithfully on the left of this ideal. 
So the question is equivalent to asking which abstract monoids are endomorphism monoids of topological spaces. (The set X, if it exists, must be the ideal of left zeroes.) This seems a hard question and I couldn't find an answer in the literature. 
De Groot proved every group is the homeomorphism group of a topological space (and the isometry group of a metric space). Apparently he asked which monoids can be the nonconstant mappings of a topological space. 

Original answer
Here is a partial answer to the second question. Among other things in Trnková, Věra
Topological spaces with prescribed nonconstant continuous mappings.
Trans. Amer. Math. Soc. 261 (1980), no. 2, 463–482 it is shown that given any monoid M, there is a topological space X such that the non-constant mappings on X form a monoid isomorphic to M. 
A: The smallest possible collection of continuous self-maps contains only the identity mapping and the constant functions.  In Groups represented by homeomorphism groups I (Mathematische Annalen, Volume 138, pages 80–102, 1959), de Groot constructs a large class of topological spaces which have this smallest possible collection as their set of continuous self-maps.
