In a nonabelian setting the correct notion of kernel is given by the kernel pair, and the correct notion of cokernel is given by the cokernel pair. For example, in any category, a morphism $f : a \to b$ is a monomorphism iff its kernel pair exists and is trivial, and dually $f$ is an epimorphism iff its cokernel pair exists and is trivial. By comparison, the naive notions of kernel and cokernel already fail to have this property for monoids in $\text{Set}$.

There are several reasonable notions of image depending on what you want to do. Let me highlight two that are particularly nice because they can be defined in terms of finite limits and colimits: the regular image $\text{im}(f)$ is the equalizer of the cokernel pair of $f$, while the regular coimage $\text{coim}(f)$ is the coequalizer of the kernel pair. (These are nonabelian generalizations of "kernel of the cokernel" and "cokernel of the kernel" respectively.)

In a category with finite limits and colimits, both of these exist and allow us to factor a morphism $f : a \to b$ as a composite

$$a \to \text{coim}(f) \to \text{im}(f) \to b$$

where the first map is a regular epimorphism and is universal with respect to that property, and the last map is a regular monomorphism and is universal with respect to that property.

Example. Let $C = \text{Top}$. Then a monomorphism and an epimorphism are just an injective and a surjective map, respectively. The kernel pair of a map $f : X \to Y$ is the pullback

$$X \times_Y X = \{ (x, y) \in X \times X : f(x) = f(y) \}$$

which one should think of as the equivalence relation on $X$ determined by $f$. Dually the cokernel pair is the pushout $Y \sqcup_X Y$. A regular monomorphism is an embedding while a regular epimorphism is a quotient map. The canonical factorization of a map $f : X \to Y$ as a composite

$$X \to \text{coim}(f) \to \text{im}(f) \to Y$$

takes the following form: $\text{coim}(f)$ is the set-theoretic image of $f$ topologized as a quotient of $X$, while $\text{im}(f)$ is the set-theoretic image of $f$ topologized as a subspace of $Y$.

The first isomorphism theorem for groups can be reintepreted in this context as the following claim:

If $f : G \to H$ is a homomorphism of groups, then $\text{coim}(f) \to \text{im}(f)$ is an isomorphism.

(The correspondence here is that $\text{im}(f)$ is the set-theoretic image while $\text{coim}(f)$ is the quotient of $G$ by the kernel of $f$. To really sell the first identification you need to first prove that every monomorphism of groups is regular, though.)

A morphism $f$ with this property is sometimes called a strict morphism. The condition that every morphism is strict, which in particular is implicitly one of the axioms defining an abelian category, is somewhat rare: for example, it already does not hold in the category of commutative rings.

Example. Let $R$ be a commutative ring and let $f : R \to S^{-1} R$ be a localization. Then the coimage-image factorization

$$R \to \text{coim}(f) \to \text{im}(f) \to S^{-1} R$$

takes the following form: the first and last maps are both identities. In particular, the middle map is just $f$ itself, so it is almost never an isomorphism.

And yet there is still a first isomorphism theorem for rings! The issue here is that the regular image doesn't agree with the set-theoretic image for rings; said another way, regular monomorphisms don't agree with monomorphisms for rings. If you like, you can consider a somewhat different generalization of the first isomorphism theorem as stated above, where you replace $\text{im}(f)$ with whatever you think the "correct" notion of image is. But $\text{coim}(f)$ is, to my mind, definitely the correct generalization of "quotient of the codomain by the kernel."

In a nonabelian setting the correct notion of kernel is given by the kernel pair, and the correct notion of cokernel is given by the cokernel pair. For example, in any category, a morphism $f : a \to b$ is a monomorphism iff its kernel pair exists and is trivial, and dually $f$ is an epimorphism iff its cokernel pair exists and is trivial. By comparison, the naive notions of kernel and cokernel already fail to have this property for monoids in $\text{Set}$.

There are several reasonable notions of image depending on what you want to do. Let me highlight two that are particularly nice because they can be defined in terms of finite limits and colimits: the regular image $\text{im}(f)$ is the equalizer of the cokernel pair of $f$, while the regular coimage $\text{coim}(f)$ is the coequalizer of the kernel pair. (These are nonabelian generalizations of "kernel of the cokernel" and "cokernel of the kernel" respectively.)

In a category with finite limits and colimits, both of these exist and allow us to factor a morphism $f : a \to b$ as a composite

$$a \to \text{coim}(f) \to \text{im}(f) \to b$$

where the first map is a regular epimorphism and is universal with respect to that property, and the last map is a regular monomorphism and is universal with respect to that property.

Example. Let $C = \text{Top}$. Then a monomorphism and an epimorphism are just an injective and a surjective map, respectively. The kernel pair of a map $f : X \to Y$ is the pullback

$$X \times_Y X = \{ (x, y) \in X \times X : f(x) = f(y) \}$$

which you should think of as the equivalence relation on $X$ determined by $f$. Dually the cokernel pair is the pushout $Y \sqcup_X Y$. A regular monomorphism is an embedding while a regular epimorphism is a quotient map. The canonical factorization of a map $f : X \to Y$ as a composite

$$X \to \text{coim}(f) \to \text{im}(f) \to Y$$

takes the following form: $\text{coim}(f)$ is the set-theoretic image of $f$ topologized as a quotient of $X$, while $\text{im}(f)$ is the set-theoretic image of $f$ topologized as a subspace of $Y$.

The first isomorphism theorem for groups can be reintepreted in this language as the following claim:

If $f : G \to H$ is a homomorphism of groups, then $\text{coim}(f) \to \text{im}(f)$ is an isomorphism.

(The correspondence here is that $\text{im}(f)$ is the set-theoretic image while $\text{coim}(f)$ is the quotient of $G$ by the kernel of $f$. To really sell the first identification you need to first prove that every monomorphism of groups is regular, though.)

A morphism $f$ with this property is sometimes called a strict morphism. The condition that every morphism is strict, which in particular is implicitly one of the axioms defining an abelian category, is somewhat rare: for example, it already does not hold in the category of commutative rings.

Example. Let $R$ be a commutative ring and let $f : R \to S^{-1} R$ be a localization. Then the coimage-image factorization

$$R \to \text{coim}(f) \to \text{im}(f) \to S^{-1} R$$

takes the following form: the first and last maps are both identities. In particular, the middle map is just $f$ itself, so it is almost never an isomorphism.

And yet there is still a first isomorphism theorem for rings! The issue here is that the regular image doesn't agree with the set-theoretic image for rings; said another way, regular monomorphisms don't agree with monomorphisms for rings. If you like, you can consider a somewhat different generalization of the first isomorphism theorem as stated above, where you replace $\text{im}(f)$ with whatever you think the "correct" notion of image is. But $\text{coim}(f)$ is, to my mind, definitely the correct generalization of "quotient of the codomain by the kernel."