When students are first learning about groups, a classic example of a group that is not defined as a set of functions is the group whose underlying set is $\mathbb{R}\setminus1$, and whose operation is $x*y=x+y+xy$. This naturally leads one to wonder about what other polynomials in two variables give rise to a group law on $\mathbb{R}$. Is there any nice criteria for such polynomials, or, in the case that there is not, are there any nice classes of polynomials that are group laws?

Emil's basic observation can be extended. Polynomials are smooth (i.e., infinitely differentiable) functions. What can you say about smooth group laws on the set of real numbers? In different language: a Lie group is a manifold which comes equipped with group operations which are smooth with respect to the manifold structure. What can you say about Lie groups whose underlying manifold is the set of real numbers with its standard manifold structure? Answer: they are all isomorphic. In particular, they are all isomorphic to the standard example. That is, there is always a smooth bijection $L$ (with smooth inverse) such that the group operation is given by $x \ast y = L^{1}(L(x) + L(y))$. This is proven in any text which treats Lie groups. So: all examples are "cheats". Of course, your example is slightly different: it is isomorphic to $(\mathbb{R}  \{0\}, \cdot)$. In fact, all Lie group structures on the punctured line are isomorphic to this one, which is $$\mathbb{Z}_2 \times \mathbb{R}$$ where the Lie group structure on $\mathbb{R}$ is, of course, given by addition. 


The only polynomial group law on $\mathbf{R}$ such that the identity element is $0$ is given by the polynomial $P(X,Y)=X+Y$. Proof. Let $P \in \mathbf{R}[X,Y]$ be the group law, with identity element $0$. For any $y \in \mathbf{R}$, let $P_y$ be the polynomial $P(X,y)$. If $y' \in \mathbf{R}$ is the inverse of $y$ with respect to the group law, then $P_y \circ P_{y'} = X$ so that $\operatorname{deg}(P_y) \cdot \operatorname{deg}(P_{y'}) = 1$. Thus the degree of $P$ with respect to $X$ is 1. Similarly the degree of $P$ with respect to $Y$ is 1. So $P=\alpha X+\beta Y + \gamma XY$. Since $P(X,0)=X$ and $P(0,Y)=Y$ then $\alpha=\beta=1$. Moreover for any $x \in \mathbf{R}$, the function $y \mapsto x+y+\gamma xy$ is a continuous bijection of $\mathbf{R}$, which forces $\gamma=0$. Note : if the identity element of $P$ is $a \in \mathbf{R}$ instead, then by considering $(X,Y) \mapsto P(X+a,Y+a)a$ we see that necessarily $P(X,Y)=X+Ya$. 

