EDITED, taking into account the commentscomments of Donu Arapura.
As JLA wrote, a homomorphism $f\colon G\to N$ gives an extension
\begin{equation}\label{e:E}
1\to K\to E\to G\to 1.\tag{E}
\end{equation}
This extension defines a homomorphism
$$b\colon G\to{\rm Out\,} K$$$$b\colon G\to \operatorname{Out} K$$
called the band (lien, kernel) of $\rm(E)$\eqref{e:E}.
By definition, $H^2(G,K,b)$ is the set of isomorphism classes of extensions $\rm(E)$\eqref{e:E} bound by $b$.
A cohomology class $\eta(E)\in H^2(G,K,b)$ is called neutral if the extension $\rm(E)$\eqref{e:E} splits, that is, there exists a homomorphism $G\to E$ such that the composite homomorphism $G\to E\to G$ is the identity automorphism of $G$. In this case we obtain an action $\varphi$ of $G$ on the normal subgroup $K$ of $E$, and we obtain an isomorphism $E$ with the semidirect product $H\rtimes_\varphi G$.
If $K$ is abelian, then ${\rm Out\,} K={\rm Aut\,}K,$$\operatorname{Out} K = \operatorname{Aut} K$, so $b$ is just an action of $G$ on $K$, and $H^2(G,K,b)$ is the usual abelian group cohomology $H^2(G,K)$, where $G$ acts on $K$ via $b$.
The set $H^2(G,K,b)$ can be described in terms of cocycles. See Section 1.14 in Springer [1][1].
The band $b$ defines an action of $G$ on the center $Z=Z(K)$, and we may consider the usual (abelian) group cohomology $H^2(G,Z)$. From the cocyclic description of $H^2(G,K,b)$ it is clear that $H^2(G,Z)$ naturally acts on $H^2(G,K,b)$.
Moreover
Moreover, if the set $H^2(G,K,b)$ is nonempty, then $H^2(G,Z)$ acts on it simply transitively; see Mac Lane, HomologyHomology, Theorem IV.8.8. The set $H^2(G,K,b)$ is nonempty if and only if a certain obstruction ${\rm Obs}(G,K,b)\in H^3(G,Z)$$\operatorname{Obs}(G,K,b)\in H^3(G,Z)$ vanishes; see Mac Lane, Theorem IV.8.7.
Note that we should not think that $H^2(G,K,b)$ "equals" $H^2(G,Z)$. First, $H^2(G,K,b)$ does not have a distinguished unit element. Secondly, $H^2(G,K,b)$ has a distinguished subset $N^2(G,K,b)$ of neutral elements. This is important because in many applications one uses nonabelian $H^2$ in order to determine whether a given extension $({\rm E})$\eqref{e:E} is split or not.
As far as I know, nonabelian $H^2$ is mostly used in the Galois cohomology setting.
Namely, if $k$ is an algebraic closure of a field $k_0$ of characteristic 0, $G={\rm Gal}(k/k_0)$$G=\operatorname{Gal}(k/k_0)$,
and $Y$ is a quasi-projective $k$-variety with additional structure (say, an algebraic group or a homogeneous space) such that for any $\sigma\in G={\rm Gal}(k/k_0)$$\sigma\in G=\operatorname{Gal}(k/k_0)$ there exists andan isomorphism $\alpha\colon\sigma Y\overset{\sim}{\to}Y$, then it defines an extension
$$1\to {\rm Aut\,} Y\to E\to G\to 1,$$$$1\to \operatorname{Aut} Y\to E\to G\to 1,$$
where $E$ is the set of such pairs $(\alpha,\sigma)$
with a suitably defined composition law.
We obtain the cohomology class $\eta(Y)\in H^2(k_0,{\rm Aut\,} Y,b)$$\eta(Y)\in H^2(k_0,\operatorname{Aut} Y,b)$
of this extension for a suitable band $b$.
The variety $Y$ (with additional structure) admits a $k_0$-model if and only if $\eta(Y)$ is neutral, that is, the extension splits;
see this question.
[1] T. A. Springer, Non-abelian $H^2$ in Galois cohomology, in: Algebraic Groups and Discontinuous
SubgroupsAlgebraic Groups and Discontinuous
Subgroups, Proc. Sympos. Pure Math. 9, Amer. Math. Soc., Providence, 1966, 164-182.
[2] M. Borovoi, Abelianization of the second nonabelian Galois cohomology. Duke Math. J. 72 (1993), 217-239Duke Math. J. 72 (1993), 217-239.
[3] Flicker, Scheiderer, Sujatha, Grothendieck's theorem on non-abelian $H^2$ and local-globallocal–global principles.
J. Amer. Math. Soc. 11 (1998), no. 3, 731–750.
See also newer papers (they refer to these three).
See also, and the preprint this preprintBorovoi - Extending the exact sequence of nonabelian $H^1$, using nonabelian $H^2$ with coefficients in crossed modules.
