This is a partial answer, summing up some of my comments.
The semi-direct product is not a limit, but rather it is a colimit. The reason is that the universal property cited above describes maps on the semi-direct product. In the special case that $\phi$ is the trivial action, the semi-direct product becomes the direct product $N \times H$ and the universal property is not just the usual universal property as a product, but rather as a representing object of the pairs of morphisms on $N,H$ which commute pointwise. In a general semi-direct product, this commutation is twisted by an action of $H$ on $N$.
So basically the idea is that we have the coproduct $N * H$ of the two groups (which is usually called the free product, which is quite unfortunate), and we impose the relation $h n h^{-1} = \phi_h(n)$. The universal property of $N \rtimes H$ is equivalent to the isomorphism
$$N \rtimes H = (N * H) / \{h n h^{-1}= \phi_h(n)\}_{h \in H, n \in N},$$
which exhibits $N \rtimes H$ as a special colimit of some diagram associated to $N,H,\phi$. However, this still uses elements in the relations. I think we cannot get rid of these elements, unless we use $2$-colimits. See below. Actually this isomorphism is used very often in group theory in order to recoqnize groups given by some presentation as a semi-direct product. For example, the dihedral group $D_n = \langle r,s : r^n = s^2 = 1, srs=r^{-1} \rangle$ is $\mathbb{Z}/n \rtimes \mathbb{Z}/2$.
On the other hand, there is a purely category-theoretic construction which is due to Grothendieck: Let $I$ be a small category and $F : I \to \mathsf{Cat}$ be a diagram of small categories. The Grothendieck construction $\int^I F$ is the category of pairs $(i,x)$, where $i$ is an object of $I$ and $x$ is an object of $F(i)$. A morphism $(i,x) \to (j,y)$ is a pair $(a,f)$, consisting of a morphism $f : i \to j$ and a morphism $a : F(f)(x) \to y$ in $F(j)$. The composition is defined by the rule
$(a_2,f_2) \circ (a_1,f_1) = (a_2 \circ F(f_2)(a_1),f_2 \circ f_1)$.
Now if $H$ is a monoid, considered as a category with just one object $*$, and $F : H \to \mathsf{Cat}$ is a diagram such that $F(*)=N$ is just a monoid, then $F$ corresponds to a homomorphism of monoids $H \to \mathrm{End}(N)$ and the Grothendieck construction $\int^H N$ has just one object, thus corresponds to a monoid, namely what is usually called the semi-direct product $N \rtimes H$. This is shown by the multiplication rule above.
Back to the general case of a diagram $F : I \to \mathsf{Cat}$, the Grothendieck construction $\int^I F$ is the lax 2-colimit of $F$. I don't know the original reference right now, but a very comprehensive account on that is the Appendix A in "The stack of microlocal sheaves" by I. Waschkies. The choice of the morphism $a : F(f)(x) \to y$ in the definition above is precisely the reason for the "2" here. If it was the identity, we would get the usual colimit.
Thus, the semi-direct product $N \rtimes H$ is the lax $2$-colimit of the diagram $N : H \to \mathrm{Cat}$.