Here is a comment about the other approach proposed in the original question, using nonabelian derived functors. It is too long to be posted as a comment, so unfortunately I have had to post it as an answer.

The category of groups is not an abelian category, but the category of modules over a fixed group G is an abelian category, and derived functors in that abelian category yield group (co)homology. We could instead directly consider the abelianization functor $ab$ from groups to abelian groups, and we could consider nonabelian derived functors of $ab$. This would involve choosing a simplicial resolution of a group $G$, applying $ab$ to that simplicial group to get a simplicial abelian group, taking the normalized Moore chain complex of $ab$, and then taking the homology groups of $ab$.

As far as I know, the standard textbook reference for these kinds of nonabelian derived functors is chapter 2 of H. Inassaridze's book "Non-abelian homological algebra and its applications." It is not a trivial matter to check that a short exact sequence of groups induces a long exact sequence of nonabelian derived functors: see Theorem 2.63 in Inassaridze's book for some sufficient conditions. In particular, I do not know if a short exact sequence of groups $1 \rightarrow K \rightarrow H \rightarrow G \rightarrow 1$ induces a long exact sequence $$\dots \rightarrow L_{n+1}ab(G) \rightarrow L_n ab(K) \rightarrow L_nab(H) \rightarrow L_nab(G)\rightarrow \dots $$ This would require checking the conditions of Inassaridze's Theorem 2.63, or something similar. In the absence of such a long exact sequence, I do not think the approach by nonabelian derived functors yields the long exact sequence $ \dots \rightarrow K_{ab} \rightarrow H_{ab}\rightarrow G_{ab} \rightarrow 1$ which you ask for.

The only positive result on nonabelian derived abelianization of groups which I was able to locate in the literature is mentioned in the proof of Theorem 3 in the paper "N-fold Cech derived functors and generalized Hopf type formulas" by G. Donadze, N. Inassaridze, and T. Porter: the first nonabelian derived functor of abelianization, $L_1ab(G)$, is apparently isomorphic to the second group homology group $H_2(G,Z)$ with integer coefficients, that is, the classical Schur multiplier of $G$.

Perhaps someone with more knowledge of nonabelian derived functors can contribute further and give a better answer to your question. It appears to me that going this route is not only more technically demanding than using group homology and the LHSSS, but there seem to be fewer tools available for making calculations (compared to the wealth of tools for computing group homology), and fewer connections to other areas of mathematics; and, possibly most troubling, it is not clear that you get the desired long exact sequence, in nonabelian derived abelianization, that you're looking for.

Here is a comment about the other approach proposed in the original question, using nonabelian derived functors. It is too long to be posted as a comment, so unfortunately I have had to post it as an answer.

The category of groups is not an abelian category, but the category of modules over a fixed group G is an abelian category, and derived functors in that abelian category yield group (co)homology. We could instead directly consider the abelianization functor $ab$ from groups to abelian groups, and we could consider nonabelian derived functors of $ab$. This would involve choosing a simplicial resolution of a group $G$, applying $ab$ to that simplicial group to get a simplicial abelian group, taking the normalized Moore chain complex of that simplicial group, and then taking the homology groups of that chain complex.

As far as I know, the standard textbook reference for these kinds of nonabelian derived functors is chapter 2 of H. Inassaridze's book "Non-abelian homological algebra and its applications." It is not a trivial matter to check that a short exact sequence of groups induces a long exact sequence of nonabelian derived functors: see Theorem 2.63 in Inassaridze's book for some sufficient conditions. In particular, I do not know if a short exact sequence of groups $1 \rightarrow K \rightarrow H \rightarrow G \rightarrow 1$ induces a long exact sequence $$\dots \rightarrow L_{n+1}ab(G) \rightarrow L_n ab(K) \rightarrow L_nab(H) \rightarrow L_nab(G)\rightarrow \dots $$ This would require checking the conditions of Inassaridze's Theorem 2.63, or something similar. In the absence of such a long exact sequence, I do not think the approach by nonabelian derived functors yields the long exact sequence $ \dots \rightarrow K_{ab} \rightarrow H_{ab}\rightarrow G_{ab} \rightarrow 1$ which you ask for.

The only positive result on nonabelian derived abelianization of groups which I was able to locate in the literature is mentioned in the proof of Theorem 3 in the paper "N-fold Cech derived functors and generalized Hopf type formulas" by G. Donadze, N. Inassaridze, and T. Porter: the first nonabelian derived functor of abelianization, $L_1ab(G)$, is apparently isomorphic to the second group homology group $H_2(G,Z)$ with integer coefficients, that is, the classical Schur multiplier of $G$.

Perhaps someone with more knowledge of nonabelian derived functors can contribute further and give a better answer to your question. It appears to me that going this route is not only more technically demanding than using group homology and the LHSSS, but there seem to be fewer tools available for making calculations (compared to the wealth of tools for computing group homology), and fewer connections to other areas of mathematics; and, possibly most troubling, it is not clear that you get the desired long exact sequence, in nonabelian derived abelianization, that you're looking for.