Let $k$ be a field, $G$ a finite group, and $k[G]$ the group algebra. Let $A$ be a subalgebra of $k[G]$. In general, $A$ is not the group algebra of some subgroup $H$ of $G$.
Question: Is there any criterion for when $A = k[H]$ for some subgroup $H$? Also, in that case, how do we read of the generating subgroup $H$? Will the situation become better/easier if I assume $A$ to be a sub-Hopf-algebra?
The characteristic of the field is important here, when considering Hopf sub-algebras. The Cartier-Kostant-Milnor-Moore theorem says that a cocommutative Hopf algebra $H$ over an algebraically closed field $k$ of characteristic 0, is a semidirect product of a group algebra and an enveloping algebra of a Lie algebra. In particular, if $H$ is finite dimensional cocommutative Hopf algebra over $k$, then $H$ is isomorphic to a group algebra.
This theorem is not true over algebraically closed fields in positive characteristics, as there are restricted enveloping $p$-Lie algebras that are finite dimensional cocommutative Hopf algebras but not isomorphic to group algebras.
See the introduction of https://cel.archives-ouvertes.fr/cel-00374383/document (by Nicolas Andruskiewitsch) for further information.
Yes, there is a criterium. Assuming $G$ finite, $A$ is of the form $k[H]$ for some subgroup of $G$ if and only if $A$ is a sub-bialgebra (and since $G$ is finite, if and only if is Hopf subalgebra).
Clearly $k[H]$ is a Hopf subalgebra of $k[G]$, but if you take any subalgebra $A$, the fact that $A$ is also a subcoalgebra means that $A$ is a subcomodule of $k[G]$, hence $G$-graded, so, the $G$-homogeneous components are $1$-dimentionals and $A$ is generated by group-like elements. But a subcoalgebra of the form $k[X]$ with $X\subseteq G$ is subalgebra only when $X$ is a subgroup.
(By the way, if $G$ is infinite everything works fine except that $H$ is maybe a submonoid and not a subgroup).
An alternative proof avoiding the grading/comodule argument is the following: $A$ being subcoalgebra means $A^*$ is an algebra quotient of $k[G]^*=k^G$= the algebra of functions from G to k. But it is clear that any quotient of $k^G$ identifies with $k^X$ with $X\subseteq G$. The rest of the argument is the same.
If $A$ is the group algebra of a subgroup, then $k[G]$ will be free as a module over $A$, and that may help to rule out some cases. This is not at all an "if and only if" statement, though.
Proposition 3.2.1 in Sweedler's book Hopf Algebras says that a cocommutative Hopf algebra is a group algebra if and only if it has a basis of group-like elements (those elements $x$ which satisfy $\Delta(x) = x \otimes x$). So if you have a sub-Hopf algebra, you can try to see if it has such a basis.
Ravenel says in Theorem 6.2.3 in Complex Cobordism and Stable Homotopy Groups of Spheres: "this is equivalent to the existence of a dual basis of idempotent elements $\{y\}$ satisfying $y_i^2=y_i$ and $y_iy_j=0$ for $i \neq j$."
