This will be a summary of the proof referenced by BCnrd in the comments, for the benefit of those that haven't looked through Katz, Mazur, Arithmetic Moduli of Elliptic Curves. A scan is available as an unsearchable djvu near the bottom of Katz's web page. I'd like to point out in particular just how different it is from Emerton's proof, and how the textbook proof I summarized in the statement of the question is essentially the same as this one, but trades some conceptual clarity for simplicity of vocabulary. The relevant corollary (12.4.6) is on page 358, which is page 185 of the scan.

Katz and Mazur start with a moduli problem $P$ (i.e., a functor from $(Ell)$ to Sets) representable by a scheme $M(P)$. We assume that $P$ is defined over an algebraic closure of $\mathbb{F}_p$, and is finite étale over $(Ell/\overline{\mathbb{F}_p})$. This means one has an étale surjection from $M(P)$ to the stack of elliptic curves over $\overline{\mathbb{F}_p}$. There is a distinguished section of the line bundle $\omega_P^{p-1}$, called the Hasse invariant. It is defined as the differential of the Verschiebung, which I will explain a bit later, and it satisfies two key properties:

1. The Hasse invariant vanishes if and only if the curve is supersingular.
2. All zeroes have multiplicity one (Igusa's theorem).

By general line bundle arithmetic, the total number of zeroes of this section in $P$, counting multiplicity, is equal to $p-1$ times the degree of $\omega$, or equivalently, $\frac{p-1}{24}$ times the degree of $P$ over (Ell).

To complete the proof, one shows that the preimage of a point under the composition $M(P) \to \mathcal{M}_{Ell} \to \mathbb{A}^1_j$ has size equal to the degree of $P$ over $(Ell)$ divided by the order of the automorphism group of the underlying elliptic curve, by using representability to deduce the freeness of the group action. This yields $$\deg P \cdot \sum_{\text{supersingular } j} \frac{1}{|\operatorname{Aut} E_j|} = \deg P \cdot \frac{p-1}{24},$$ and we are done.

The Verschiebung and the Hasse invariant deserve some additional explanation. For any $\mathbb{F}_p$-scheme $X$, there is an absolute Frobenius map $X \to X$ which on affines is the functor $\operatorname{Spec}$ applied to the $p$-th power map. For an $\mathbb{F}_p$-scheme $S$ and an $S$-scheme $X$, one obtains an $S$-scheme $X^{(p)}$ as the pullback of $X$ over the absolute Frobenius on $S$. By the universal property of pullbacks, the absolute Frobenius on $X$ factors through an $S$-map $X \to X^{(p)}$, called the relative Frobenius. Over an elliptic curve $E \to S$, this map turns out to be an isogeny of degree $p$. The Verschiebung $V: E^{(p)} \to E$ is defined as the dual to the relative Frobenius isogeny $F: E \to E^{(p)}$, and the multiplication by $p$ map on $E$ factors as $[p] = VF$.

The kernel of Frobenius is always a connected group scheme of length $p$, and the kernel of the Verschiebung over a field is connected if and only if the full $p$-torsion subgroup is connected. The latter case can be taken as a definition of supersingularity over geometric points, and for general $\mathbb{F}_p$-schemes, any family of elliptic curves with at least one supersingular geometric fiber is called supersingular. The Hasse invariant is the induced map on $S$-Lie algebras: $\text{Lie}V: \text{Lie}(E/S)^{(p)} \to \text{Lie}(E/S)$. It is an isomorphism if and only if $E$ is not supersingular (i.e., $E$ is ordinary), as one can calculate by examining formal groups. We can write it as an element of $\text{Hom}(\text{Lie}(E/S)^{(p)}, \text{Lie}(E/S)) = \text{Hom}(\text{Lie}(E/S)^{\otimes p}, \text{Lie}(E/S)) = H^0(S, \omega_{E/S}^{\otimes p-1})$. It is a modular form of weight $p-1$, and its $q$-expansion is identically 1.

The fact that $\omega_P$ has degree $\frac{1}{24}\deg P$ is still a bit of a conceptual mystery to me. It is proved by first calculating it for a full level 3 structure (which has degree 24) and then transferring to all other representable moduli problems.