One defines the $H^n(G,M)$ where $M$ is a $\mathbb{Z}[G]$ module as $Ext^n_{\mathbb{Z}[G]}(\mathbb{Z},M)$ where $\mathbb{Z}$ is viewed as a trivial $\mathbb{Z}[G]$-module.
Is this part of a general pattern for how to define cohomology for non-abelian categories?
In groups we see that we switch to the abelian category of $\mathbb{Z}[G]$-modules. If this idea is indeed extended for general non-abelian categories, what abelian category do we switch to?
Yes, this generalizes to Hochschild cohomology and André-Quillen cohomology.
Given a category $\mathcal{C}$ with finite limits and an object $X$, you can form the category of Beck modules $\operatorname{Ab}(\mathcal{C} / X)$, abelian group objects in the slice category over $X$. When $\mathcal{C}$ is the category of groups, a Beck module over $G$ is precisely a split extension of $G$ with abelian kernel, so Beck modules can be identified with $\mathbb{Z}[G]$-modules.
In nice cases, the category $\operatorname{Ab}(\mathcal{C} / X)$ is abelian, and the forgetful functor $\operatorname{Ab}(\mathcal{C} / X) \to \mathcal{C} / X$ has a left adjoint, called abelianization $\operatorname{Ab}_X$. The Hochschild cohomology of a Beck module $M$ is defined to be $HH^i(X; M) = \operatorname{Ext}^i(\operatorname{Ab}_X X, M)$. In the groups case, $\operatorname{Ab}_G G$ is the augmentation ideal, so Hochschild cohomology is just group cohomology shifted by 1.
Hochschild cohomology itself is only an approximation to André-Quillen cohomology, which is somehow a more homotopically correct notion (I don't know enough about any of this to explain what this means, exactly). In the case of groups, these two cohomology theories coincide. I suggest having a look at Martin Frankland's thesis, which gives a good overview of these things. It works out explicit examples (groups, abelian groups, associative algebras, commutative algebras) very nicely in Appendix A.
