Well, this might not be the answer you expected:
In general, there is no coproduct such that $Ext_H^\ast(k,k)$ (cup product) is a graded Hopf algeba.
For, let $k$ be a perfect field and suppose $A = Ext_H^\ast(k,k)$ is a graded Hopf algebra of finite type. By Borel's structure theorem on connected graded commutative Hopf algebras [A-M, VI.2.8], $A$ is (as $k$-algebra) isomorphic
to the tensor product of algebras of the types $k[x]$ and $k[y]/(y^r)$. In particular, $A/rad(A)$ is a domain.
Now it's easy to find counterexamples. For instance let $\text{char}(k)=2$. Then the cohomology ring of the dihedral group $H^\ast(D_8;k) = k[x,y,z]/(xy),\;|x|=|y|=1, |z|=2$ has zero-divisors, but its radical is zero.
More generally: If $\text{char}(k) = p$ and $G$ is a $p$-group with at least two conjugacy classes of maximal elementary abelian subgroups, then there are non-nilpotent classes $x,y \in H^\ast(G;k)$ such that $xy=0$. Thus $Ext_{k[G]}^\ast(k,k) = H^\ast(G;k)$ can't be a Hopf algebra.
However, if $H$ is commutative, then the product induces a coproduct that makes $Ext_H^\ast(k,k)$ a commutative, cocommutative Hopf algebra. I guess searching for a reference will probably last much longer than the straightforward proof: Let $P \to k$ be a projective resolution over $H$. Then the operations on $H$ and the uniqueness of induced mappings (up to homotopy) induce a (kind of) DG-Hopf algebra $(P,\mu,\Delta)$ where the usual diagramms commute up to homotopy. Passage to cohomology now makes the diagramms commute and you have your Hopf algebra.
Note: $(P,\mu,\Delta)$ can be seen as an algebraic $H$-space analog.
One may wonder why the coproduct $\Delta: H \to H \otimes H$ always induces a product on $Ext_H^\ast(k,k)$ while the product $\mu: H \otimes H \to H$ induces a coproduct only if $H$ is commutative. The reason is that $\Delta$ is an algebra homomorphism while $\mu$ is an algebra homomorphism if and only if $H$ is commutative.
[A-M] Adem, Milgram: Cohomology of finite groups.