To answer the question, putting together points made in the comments...

There is always a set of functions between two sets, unless you are using predicative foundations, where there may be no function sets, or the set theory NF(U) which is classical (even material!) but famously has no function sets.

However, this is not what the authors of the article are considering, rather they are considering (in ordinary classical foundations) a disjoint union
$$
\coprod_{L\in BigIntervals} Top_\ast(L,X)
$$
where each $Top_\ast(L,X)$ is a set. For Hausdorff $X$, the authors state

"We will be saved from set-theoretic difﬁculties only by the remarkable fact that all maps deﬁned on big intervals into a ﬁxed space $X$ factor through maps deﬁned on relatively small big intervals."

Thus they are considering the quotient of the above disjoint union by a relation 'factors through'. That is, for a fixed $X$, there is a *set* $\{L_\alpha\}_{\alpha\in I}$ of big intervals such that every $f\colon L\to X$ for an arbitrary big interval $L$ factors through some $f_\alpha\colon L_\alpha \to X$, and then $f \sim f_\alpha$. That this isn't a problem is resolved by some variant of Scott's trick, which allows for taking quotients of proper classes, or by arguing directly that the above disjoint union should be replaced by
$$
\coprod_{L_\alpha\in I} Top_\ast(L_\alpha,X).
$$

In the event that one really has a proper class, then the fact that there is a group structure (class function giving multiplication etc) on that class means that it is a group, just not a set-sized group. Some people (e.g. Conway) would call this a Group, to distinguish it; category theorists might call it a 'large group', if using the terminology of 'small' and 'large' to distinguish set-theoretic size issues.