Here's a repair of the definition suggested in the OP. Namely, we need the additional assumption that $\text{Hom}(1, -) : C \to \text{Set}$ is faithful (so concretizes $C$). Then with the condition described in the OP, we get a map of sets $^{-1} : \text{Hom}(1, G) \to \text{Hom}(1, G)$, and now it really makes sense to ask whether this lifts to a morphism in $C$; faithfulness ensures that if such a morphism exists, it is unique, and therefore really a property of $G$.
So it may happen that $^{-1}$ exists but is not a morphism in $C$ (and that this is the sense in which $G$ is a group object), but also that $\text{Hom}(1, -)$ factors through some other category $D$ in which it is a morphism. For example, if $C$ is the category of Poisson manifolds, then apparently $D$ is the category of smooth manifolds. This provides some motivation for the definition in Ryan Reich's answer.