No, let $G$ be an arbitrary group and take $H=G$ acting on itself by conjugation. Then $H$ be comes the diagonal in $G\rtimes H\cong G\times G$. This is well known to be a Gelfand pair.