I have seen this statement about $E_2^*$ tossed around off-handedly by experts a number of times, but never seen a complete proof referenced.
The tools to prove it are (mostly) in Masser's "Elliptic functions and transcendence", Appendix 1. There, Masser gives a formula for certain non-holomorphic modular functions. One of these fomrulas (Lemma A3) can be re-written as $$E_2^*(\tau)\left(\frac{\pi}{\omega_1}\right)^2=-\frac{3S}{\sqrt{D} ~\tau},$$ where $(\omega_1,\omega_2)$ are choices of periods of a CM elliptic curve with rational equation, $D$ is the discriminant of the CM point $\tau=\frac{\omega_1}{\omega_2}$, and $S$ is a sum of division points on the curve. If $\tau$ satisfies the reduced, integral quadratic, $C\tau^2+B\tau+A$, then Masser points out that by a theorem of Baker, $(AC)^2\wp$ is an algebraic integer. Moreover, the norm of $\tau$ is $A/C$, and so it's clear the only additional primes that could divide the denominator are divisors of $AC$.
On page 118 of Masser, he offers formulas for the function $$\gamma(\tau)=\frac{E_2^*(\tau)E_4(\tau)}{6E_6(\tau) j(\tau)}-\frac{7j(\tau)-6912}{6j(\tau)(j(\tau)-1728)}$$ at CM points, in terms of singular moduli of $j(\tau)$. The Gross-Zagier formula can then be used to show that no primes that split in the CM field can divide the denominator. Since any prime dividing $A$ or $C$ must split in the CM field, none of these primes can divide the denominators, and so $\sqrt{D}E_2^*(\tau)\left(\frac{\pi}{\omega_1}\right)^2$ must be an algebraic integer.