UPDATE: Ok, here is some data.

$$q^{-1} = j - 744 - 196884/j + \cdots = \mathbb{Z} - 7.5 \times 10^{-13} + \cdots$$ $$q^{-2} = j^2 - 1488 j + 159768 - 42987520 / j + \cdots = \mathbb{Z} - 1.6 \times 10^{-10}$$

From here, it get's too long to write out the details, so I'll just give the first noninteger term:

$$q^{-3} = \mathbb{Z} -9.88 \times 10^{-9} + \cdots$$$$q^{-4} = \mathbb{Z} -3.08 \times 10^{-7} + \cdots$$$$q^{-5} = \mathbb{Z}-6.35 \times 10^{-6} + \cdots$$$$q^{-6} = \mathbb{Z}-9.72 \times 10^{-5} + \cdots$$$$q^{-7} = \mathbb{Z}-1.19 \times 10^{-3} + \cdots$$$$q^{-8} = \mathbb{Z}-1.22 \times 10^{-2} + \cdots$$$$q^{-9} = \mathbb{Z}-0.109 + \cdots$$$$q^{-10} = \mathbb{Z}-0.860 + \cdots$$

Juding from Michael's data above, the later terms also make substantial contributions, but I think this explains a large part of the mystery.

And, in case it is useful to anyone, here is the first 14 coefficients of $q^{-1}$ as a power series in $j^{-1}$.

{1, -744, -196884, -167975456, -180592706130, -217940004309744, -282054965806724344, -382591095354251539392, -536797252082856840544683, -772598111838972001258770120, -1134346327935015067651297762308, -1692324738742597705005194275401888}

Notice that the series starts $q^{-1} = j - 744 -196884/j + \cdots$.

1

There is an obvious computation here that no one seems to be doing. Invert the power series for $j$, to write

$$q^{-1} = j + a(0) + a(1) j^{-1} + a(2) j^{-2} + \cdots.$$

(Notice that the $a(i)$ will be integers.)

Raise this to the, for example, $5$th power to get

$$q^{-5} = j^5 + b(-4) j^4 + \cdots + b(0) + b(1) j^{-1} + \cdots.$$

See whether the coefficients $b$ are dropping off fast enough to give a simple explanation. People are assuming that the fact that the coefficients for $j$ in terms of $q$ grow rapidly means that the coefficients for the inverse will grow rapidly too, but that doesn't seem justified to me.