A reference is:
D.F. Holt, The calculation of the Schur multiplier of a permutation group. In: Michael D. Atkinson, Edotor, Computational Group Theory (Conference proceedings, Durham, 1982), Academic Press, 1984, pages 307-319.
But you might have difficulty finding it!
The idea of this algorithm is to find the $p$-parts of the multiplier separately for the primes $p$ dividing $|G|$. To do that, we first find the multiplier $M(P)$ of a Sylow $p$-subgroup $P$ of $G$ using part of the $p$-quotient algorithm, and then find the $M(G)_p$ as the subgroup of $G$-stable elements of $G$. (I programmed this myself first in ALGOL60 and then in C in the early 1980s, partly motivated by the fact that there had been so many errors in the calculation of the multipliers of the finite simple groups - it had taken three attempts to get $M(M_{22})$ right!)
There is a much simpler algorithm available in Magma as $\mathtt{Darstellungsgruppe}$ that takes as input a finite presentation $\langle X \mid R \rangle$ of the finite group $G$, and finds a presentation of a Schur-covering group $C(G)$ of $G$ by factoring out a free abelian subgroup of $R/[F(X),R]$ in $F(X)/[F(X),R]$ using the Hopf formula. The multipler can then be calculated as the kernel of the natural map $C(G) \to G$. This works OK but only for moderately small groups $G$. Here is an example with $G=A_5$.
> G:=Group<x,y|x^2,y^3,(x*y)^5>;
> C,phi:=Darstellungsgruppe(G);
> K:=Kernel(phi);
> #K;
2