Take $C$ to be the category of dualizable $N$-$N$-bimoduls, $N$ a factor. A subfactor $N\subset M$ 
(or $N_0\subset N$) with finite index and finite depth gives an algebra object $A$ in $C$, namely $A={}_NM_N$ (or ${}_NL^2M_N$ if you prefer) and conversely an algebra object (more precisely a Q-system) gives a subfactor $N\subset M$. Instead of building artificially a category of subfactors, you take the category of (simple) algebra objects (Q-systems) in $C$.
Each finite group gives an object
$A_G=\bigoplus_{g\in G} {}_NN^{\circ\alpha_g}_N$ with $\alpha_g$ automorphisms on $N$ such that $\alpha_g\alpha_h=\alpha_{gh}$
 for $g,h\in G$ and ${}_NN^{\circ\alpha_g}_N$ is ${}_NN_N$ seen as a $N$-$N$ bimodule, where the right action is composed with $\alpha$. A morphism $H\to G$ gives a morphism $A_H\to A_G$ between algebra object. This category contains also finite groups, their duals, Kac-algebras and weak-C${}^\ast$ Hopf algebras.

If you want irreducible subfactors, you ask $A$ to be haploid, then you lose weak-C${}^\ast$ Hopf algebras.

This also tells you how a "category of subfactors" should work...