I think to complete David White's answer it is also interesting to mention some stuff :
A group in a category $\mathcal{C}$ with products is as David said an object with some data involving only finite products.
One can say cogroup object in a category $\mathcal{C}$ with finite coproducts is a group object in $\mathcal{C}^{\operatorname{op}}$ and you get David's description with coproducts.
(With these definitions, you can see that if $G$ is a group, then for any $X$, $\operatorname{Hom}(X,G)$ is a group in $Set$ and if $G$ is a cogroup, then for any $X$, $\operatorname{Hom}(G,X)$ is a group in $Set$, this will be relevant for an example later)
But I think a better setting would be to fix a unital monoidal category $(\mathcal{C},\otimes,\mathbf{1})$ and to say that a group object is the data of an object $G$ with maps $G \otimes G \rightarrow G$, $\mathbf{1} \rightarrow G$, and $G\rightarrow G$ called respectively the mulitplication, the unit, the inverse and such that they make some diagram commute (corresponding to the unit, the associativity, and the inverse axioms) as in the nlab page. For instance taking group objects in $(Set,\times,*)$ you recover the classical notion of groups.
A group action of a group object $G$ on $X$ would be the data of a map $G\otimes X \rightarrow X$ satisfying some diagrams that commutes provided on the nlab page. Again for $Set$ you recover the classical notion of actions.
Once this settled up, you can say that a cogroup object in $(\mathcal{C},\otimes,\mathbf{1})$ is a group object in $(\mathcal{C}^{\operatorname{op}},\otimes,\mathbf{1})$ with the same monoidal structure, and a cogroup action is a group action in $(\mathcal{C}^{\operatorname{op}},\otimes,\mathbf{1})$. This does not fit in the first approach of cogroup I have given but it is more general because in that setting you can talk about groups/cogroups for monoidal structures other that the cartesians one. You can write what it is explicitely by taking the axioms of a group object and taking the opposite of all arrows.
Before giving some example I have a little warning to make. In a closed monoidal category, the tensor product $\otimes$ has a right adjoint $\mathcal{Hom}$ called the internal Hom. For instance in $Set$ the Hom set in the internal Hom. I think you can prove (maybe with some extra assumptions) that $\mathcal{Hom}(X,X)$ is always a monoid object (the same as a group object but you don't ask for an inverse map). In that case, for group actions, you can dualize the data $G\otimes X \rightarrow X$ into a data $G \rightarrow \mathcal{Hom}(X,X)$. Asking for $G \otimes X \rightarrow X$ to satisfy action axioms is exactly the same as asking $G \rightarrow \mathcal{Hom}(X,X)$ to be a monoid homomorphism. And you recover the definition of group action you were referring to.
But for cogroups, in general you can't say $\mathcal{Hom}$ is left adjoint to $\otimes$ so you can't really talk about cogroups that way (or at least I don't know any good examples of object being "cogroups" using the $\mathcal{Hom}$ characterization or with a left adjoint to $\otimes$ characterization).
Here are some examples of cogroups :
Hopf algebras are special case of cogroups in the category of unital rings.
In $(hTop_*,\vee)$ (Pointed topological spaces with maps between homotopy classes of pointed continuous maps, and $\vee$ the smash product), the spheres $S^n$ with a fixed points are cogroups. The counit $S^n \rightarrow *$ being the terminal map, the coinverse is the symmetry with the equator $S^{n-1} \subset S^n$ and the comulitplication is $S^n \rightarrow S^n \vee S^n$ is the pinching operator sending $S^n$ to $S^n/S^{n-1} \simeq S^n \vee S^n$. You can moreover see that for $(X,x)$ a pointed topological space, $\operatorname{Hom}_{hTop_*}(S^n,(X,x)) = [S^n,(X,x)]$ is a group. In fact, that's a way to define $\pi_n(X,x)$.
For coactions maybe you can first see that monoid objects in $\mathbb{Z}-Mod$ with the tensor product, are rings. And (left) actions of rings are left modules over rings. If $G$ is a finite group, I think the function ring $\mathcal{C}(G,\mathbb{R})$ would be a (commutative) Hopf algebra, and in particular a cogroup in $\mathbb{R}-Mod$. And modules that have a coaction on this are exactly representations of $G$.