No, let $G$ be an arbitrary group and take $H=G$ acting on itself by conjugation. Then $H$ becomes the diagonal in $G\rtimes H\cong G\times G$. This is well known to be a Gelfand pair.
PS: The Hecke algebra of the pair $(G\rtimes H,H)$ is in fact equal to the fixed point algebra $\mathbb C[G]^H$. This explains your observation when $G$ is abelian.