I am recording here a "regularized" version of the standard contour integral argument that avoids detours (though it still needs the standard contour traversing the standard fundamental domain). I first need a regularized version of the residue theorem:
Smoothed residue theorem. If $f: U \to {\bf C}$ is holomorphic and not identically zero on some connected open $U$, and $\psi \in C^\infty_c(U)$, then
$$ \sum_\rho \psi(\rho) = -\frac{1}{\pi} \int_U \frac{f'}{f}(x+iy) \partial_{\bar z} \psi(x+iy)\ dx dy$$
where $\partial_{\bar z} = \frac{1}{2} (\partial_x + i \partial_y)$ is the Wirtinger derivative, and $\rho$ ranges over the zeroes of $f$ counted with multiplicity.
Proof We can factor $f(z) = g(z) \prod_\rho (z-\rho)$ where $\rho$ ranges over the zeroes of $f$ in the support of $\psi$ (counting multiplicity), and $g$ is non-vanishing on $\psi$. Then $\frac{f'}{f}(z) = \frac{g'}{g}(z) + \sum_\rho \frac{1}{z-\rho}$, and $\partial_{\bar z} \frac{g'}{g}=0$ by the Cauchy-Riemann equations, so it suffices to show that
$$ \psi(\rho) = -\frac{1}{\pi} \int_U \frac{1}{x+iy-\rho} \partial_{\bar z} \psi(x+iy)\ dx dy.$$
Using polar coordinates $x+iy = \rho + re^{i\theta}$, the right-hand side is
$$ -\frac{1}{2\pi} \int_0^\infty \int_0^{2\pi} \frac{1}{re^{i\theta}} (e^{i\theta} \partial_r + ie^{i\theta} \partial_\theta) \psi(\rho+re^{i\theta}) \ r dr d\theta,$$
and the claim then follows from two applications of the fundamental theorem of calculus. $\Box$
Now we work on the uncompactified modular curve $Y(1) = \Gamma \backslash {\bf H}$ with $\Gamma := \mathrm{SL}_2({\bf Z})$, which has the standard fundamental domain $F := \{ z \in {\bf H}: |z| \geq 1, |\mathrm{Re} z| \leq 1/2\}$:
Smoothed valence formula for Y(1). If $f$ is a modular form of weight $k$ that is not identically zero, and $\psi \in C^\infty_c(Y(1))$, then
$$ \sum_{\Gamma \rho} \psi(\Gamma \rho)/|\mathrm{Stab}(\rho)| = - \frac{1}{\pi} \int_F \frac{f'}{f}(x+iy) \partial_{\bar z} \psi(x+iy)\ dx dy + \frac{ik}{2\pi} \int_{\pi/3}^{\pi/2} \psi(e^{i\theta})\ d\theta,$$
where $\Gamma \rho$ ranges over the elements of $Y(1)$ on which $f$ vanishes (counting multiplicity), and $\mathrm{Stab}(\rho)$ is the stabilizer of the $\Gamma$ action on $\rho$ (thus this is of order two if $\Gamma \rho = \Gamma i$, of order three if $\Gamma \rho = \Gamma e^{\pi i/3}$, and trivial otherwise).
Proof. Using a smooth partition of unity (to localize the support of $\psi$ to individual coordinate charts) we see that we can write
$$ \psi(\Gamma z) = \sum_{\gamma \in \Gamma} \varphi(\gamma z)$$
for some $\varphi \in C^\infty_c(\mathbf{H})$. Then we can unfold
$$ \sum_{\Gamma \rho} \psi(\Gamma \rho)/|\mathrm{Stab}(\rho)| = \sum_\rho \varphi(\rho)$$
so by the smoothed residue theorem it suffices to show that
$$ \int_{\bf H} \frac{f'}{f}(x+iy) \partial_{\bar z} \varphi(x+iy)\ dx dy = \int_F \frac{f'}{f}(x+iy) \partial_{\bar z} \psi(x+iy)\ dx dy - \frac{ik}{2} \int_{\pi/3}^{\pi/2} \psi(e^{i\theta})\ d\theta.$$
We can split the left-hand side using the fundamental domain into
$$ \sum_{\gamma \in \Gamma} \int_{\gamma F} \frac{f'}{f}(x+iy) \partial_{\bar z} \varphi(x+iy)\ dx dy $$
which after a change of variables becomes
$$ \sum_{\gamma \in \Gamma} \int_{F} \frac{f'}{f}(\gamma z) (\partial_{\bar z} \varphi)(\gamma z)\ \frac{dx dy}{|cz+d|^4}$$
where we write $z=x+iy$ and $\gamma z= \frac{az+b}{cz+d}$. From modularity we can compute
$$ \frac{f'}{f}(\gamma z) = (cz+d)^2 \frac{f'}{f}(z) + k c(cz+d)$$
while from the chain rule we have
$$ (\partial_{\bar z} \varphi)(\gamma z) = (\overline{cz+d})^2 \partial_{\bar z} (\varphi(\gamma z)).$$
The contributions involving $\frac{f'}{f}$ now line up, and it remains to show that
$$ \sum_{\gamma \in \Gamma} \int_{F} \frac{c}{cz+d} (\partial_{\bar z} \varphi(\gamma z))\ dx dy = - \frac{i}{2} \int_{\pi/3}^{\pi/2} \psi(e^{i\theta})\ d\theta.$$
Since $\partial_{\bar z} \frac{c}{cz+d} = 0$ (and $\varphi$ is compactly supported), we can apply Stokes' theorem and the definition of the Wirtinger derivative to write the left-hand side as
$$ \frac{-i}{2} \sum_{\gamma \in \Gamma} \int_{\partial F} \frac{c}{cz+d} \varphi(\gamma z)\ dz$$
where $\partial F$ is traversed anticlockwise.
The contribution of the vertical sides of $\partial F$ can be combined as
$$ \frac{-i}{2} \int_{e^{i\pi/3}}^{e^{i\pi/3}+i\infty} \sum_{\gamma \in \Gamma} (\frac{c}{cz+d} \varphi(\gamma z) - \frac{c}{c\alpha z+d} \varphi(\gamma \alpha z))\ dz$$
where $\alpha$ is the shift $z \mapsto z-1$. Relabeling $\gamma \alpha$ by $\gamma$ in the second term (which changes $\frac{c}{c\alpha z+d}$ to $\frac{c}{cz+d}$), we see that this expression cancels to zero. Similarly, the contribution of the bottom side of $\partial F$ is
$$ \frac{-i}{2} \int_C \sum_{\gamma \in \Gamma} (\frac{c}{cz+d} \varphi(\gamma z) - \frac{1}{z^2} \frac{c}{c\beta z+d} \varphi(\gamma \beta z))\ dz$$
where $\beta$ is the inversion map $z \mapsto -1/z$, and $C$ is the anticlockwise contour on the circle from $e^{\pi i/3}$ to $i$. Relabeling $\gamma \beta$ to $\gamma$ in the second term (which changes $\frac{c}{c\beta z+d}$ to $\frac{d}{-d/z-c}$), the claim then follows after a routine calculation. $\Box$
Now we pass to the compactification $X(1) = Y(1) \cup \{\infty\}$.
Smoothed valence formula for X(1). If $f$ is a modular form of weight $k$ that is not identically zero, and vanishes to order $m$ at infinity, and $\psi \in C^\infty(X(1))$, then
$$ \sum_{\Gamma \rho} \psi(\Gamma \rho)/|\mathrm{Stab}(\rho)| + m \psi(\infty) = - \frac{1}{\pi} \int_F \frac{f'}{f}(x+iy) \partial_{\bar z} \psi(x+iy)\ dx dy + \frac{ik}{2\pi} \int_{\pi/3}^{\pi/2} \psi(e^{i\theta})\ d\theta.$$
Proof In view of the smoothed valence formula for $Y(1)$, we may assume that $\psi$ is supported on a neighborhood of infinity that avoids all other zeroes of $f$. Our task is now to show that
$$ \int_F \frac{f'}{f}(x+iy) \partial_{\bar z} \psi(x+iy)\ dx dy = - \pi m.$$
Since $(f'/f) \psi$ decays (exponentially fast) to $2\pi i m \psi(\infty)$ and obeys the Cauchy-Riemann equations $\partial_{\bar z} \frac{f'}{f} = 0$, this follows from a routine integration by parts and a limiting argument. $\Box$
Finally, taking $\psi=1$ in the smoothed valence formula for $X(1)$, we recover the classical valence formula.
