What monsters does the "growth condition" required of holomorphic modular functions bar? Even though the title of this question pretty much captures what I'd like to know, I'll add
two side questions:
1)  Is it difficult to get a handle on the totality of functions that arise if one drops
the growth condition?
2)  Presumably these functions have no known number theoretic interest; is there a theoretical reason to explain why they shouldn't?
Edit:  In light of the early responses (thank you all!), particularly Scott Carnahan's, I'm particularly interested to know about modular functions with essential singularities and I suppose non-zero weight.  As Scott points out, for quite arbitrary $g$, $g(f(z))$ will be $\Gamma$-invariant if $f$ is $\Gamma$-invariant ($\Gamma\subset SL_2{\Bbb Z}$).  Then
I suppose one could multiply such a $g(f(z))$ by a classical modular function of
non-zero weight.  
Does this observation admit some sort of converse, something in the spirit of the Weierstrass factorization theorem for entire functions, say?   
 A: There are several growth conditions at infinity that you can impose:


*

*If $f(z) \to 0$ as $z \to i \infty$ (or in higher level, all cusps), then $f$ is a cusp form, and if it is weight $2k$, then it descends to a section of $\Omega^{\otimes k}$ on the compact modular curve.

*If $f(z)$ is bounded as $z \to i \infty$ (this is the condition you mention in the comments), then $f$ is often called a holomorphic modular form, and it descends to a section of $\Omega^{\otimes k}(\text{cusps})$, i.e., these are pluricanonical forms with logarithmic poles at cusps.

*If $f(z)$ grows at most exponentially as $z \to i \infty$, then we can call $f$ a meromorphic modular form, and it descends to a section of $\Omega^{\otimes k}(\infty  \cdot \text{cusps})$.  Common examples of number theoretic interest include the $j$-invariant, other principal moduli of genus zero modular curves, partition functions of conformal vertex algebras, and forms of negative weight that enumerate colored partitions.  You might want to rephrase your claim of "no number theoretic interest" to a statement that there is no obvious connection to Galois representations through Langlands correspondence.

*If $f(z)$ has no growth condition at all, then it does not necessarily descend to a section of a bundle on an algebraic curve, and it only exists as an analytic object.  For example, you can pull back the exponential function by $j$ to get the holomorphic $SL_2(\mathbb{Z})$-invariant function $e^{j(z)}$ on the upper half-plane that grows like $e^{e^z}$ as $z \to i\infty$ along some trajectories.  I don't know of any number theoretic applications for modular functions with essential singularities at cusps.
A: The coefficients of functions with poles at cusps do  have great number theoretic interest, and have been heavily studied. A classical example is the coefficients of $1/\eta(\tau)$, whose coefficients are partitions of a number. It is also fairly easy to describe all such functions that are meromorphic at cusps: these are more or less just sections of various line bundles over a compact Riemann surface, so the number of such functions is given by Riemann-Roch, just as for modular forms that are bounded at cusps. 
