In response to your first question: yes, the theory of extensions with non-abelian kernel has been developed, by Eilenberg, Mac Lane and others. It is summarized in Section IV.6 of Ken Brown's "Cohomology of groups" book, where you will find references to the original literature.

When the kernel $K$ is non-abelian, as you rightly say there is no conjugation action of $H$ on $K$, only an outer action, ie a homomorphism $\psi\colon\thinspace H\to \mathrm{Out}(K) := \mathrm{Aut}(K)/\mathrm{Inn}(K)$. Then extensions which induce this outer action are classified up to equivalence by $H^2(H;ZK)$, where $ZK$ denotes the center of $K$.

See also the reference linked to in Peter Michor's answer to my recent question, which may contain more relevant information.

In response to your first question: yes, the theory of extensions with non-abelian kernel has been developed, by Eilenberg, Mac Lane and others. It is summarized in Section IV.6 of Ken Brown's "Cohomology of groups" book, where you will find references to the original literature.

When the kernel $K$ is non-abelian, as you rightly say there is no conjugation action of $H$ on $K$, only an outer action, ie a homomorphism $\psi\colon\thinspace H\to \mathrm{Out}(K) := \mathrm{Aut}(K)/\mathrm{Inn}(K)$. Then extensions which induce this outer action are classified up to equivalence by $H^2(H;ZK)$, where $ZK$ denotes the center of $K$.

See also the reference linked to in Peter Michor's answer to my recent question, which may contain more relevant information.