The group $\Gamma$ is indeed isomorphic to the Galois groups of the fields in the family, whose class groups one studies. The class groups come with a natural action of $\Gamma$, but under this action, the primitive central idempotent $\sum_{g\in \Gamma}g$, which corresponds to the trivial representation, acts as zero on the coprime-to-$|\Gamma|$ part (in other words, the coprime-to-$|\Gamma|$ part has no fixed points under the action of $\Gamma$), so this part is naturally a module under the quotient ring $\mathbb{Z}[\Gamma]/\langle\sum_{g\in \Gamma}g\rangle$ (and more, see below).

In general, for arbitrary finite groups $\Gamma$, $\mathbb{Q}[\Gamma]$ is a semisimple $\mathbb{Q}$-algebra, i.e. a direct product of matrix algebras over division algebras. It always has a direct summand isomorphic to $\mathbb{Q}$, generated by the above-mentioned idempotent, and Cohen–Lenstra pass to the quotient, since that idempotent is not interfering with their Galois action.

If $R$ is any integrally closed Noetherian domain (such as $\mathbb{Z}$, or the localisation $\mathbb{Z}_{(p)}$ at a prime $p$), and $\mathbb{K}$ is the field of fractions, then an $R$-order in a semisimple $\mathbb{K}$-algebra is a subring that is a finitely generated $R$-module and that generates the algebra over $\mathbb{K}$. For example $\mathbb{Z}[\Gamma]$ is a $\mathbb{Z}$-order (or just "order" when $R=\mathbb{Z}$) in $\mathbb{Q}[\Gamma]$. But also, $\mathbb{Z}_{(p)}[\Gamma]$ is a $\mathbb{Z}_{(p)}$-order in the same algebra. A maximal $R$-order is an $R$-order that is not properly contained in any other $R$-order. It requires proof that such a thing exists. In the particular case that Cohen–Lenstra are talking about, the maximal order is unique, and contains $\mathbb{Z}[\Gamma]$.

One can show by localising that for any $\Gamma$ (not just abelian), if $A_\Gamma$ is a maximal order in $\mathbb{Q}[\Gamma]$ that contains $\mathbb{Z}[\Gamma]$, then the coefficients of the elements in $A_{\Gamma}$ have denominators supported only at prime divisors of $|\Gamma|$. In other words, if $p$ is a prime that does not divide $|\Gamma|$, then $\mathbb{Z}_{(p)}[\Gamma]$ is a maximal $\mathbb{Z}_{(p)}$-order in $\mathbb{Q}[\Gamma]$. It follows that if $A$ is a $\Gamma$-module of order coprime to $|\Gamma|$, then $A$ has a well-defined action of $A_{\Gamma}$ (because if $p\nmid |A|$, then you can "divide by $p$" in $A$).

For this and much more on maximal orders, see I. Reiner's book "Maximal Orders".

When $\Gamma$ is not assumed to be commutative, a maximal order in $\mathbb{Q}[G]$ is no longer unique, in general, and the situation is more complicated. This case was treated a few years later by Cohen–Martinet.