When students are first learning about groups, a classic example of a group that is not defined as a set of functions is the group whose underlying set is $\mathbb{R}\setminus1$, and whose operation is $x*y=x+y+xy$. This naturally leads one to wonder about what other polynomials in two variables give rise to a group law on $\mathbb{R}$. Is there any nice criteria for such polynomials, or, in the case that there is not, are there any nice classes of polynomials that are group laws?

4$\begingroup$ Your operation only defines a group on $\mathbb R\smallsetminus\{1\}$. It can be generalized to $x∗y=L^{−1}(L(x)L(y))$ on $\mathbb R\smallsetminus\{L^{1}(0)\}$, where $L$ is any nonconstant linear function. This also shows that the example is a cheat, the group is just ordinary multiplication, except that the domain has been shifted. $\endgroup$ – Emil Jeřábek Mar 11 '11 at 13:05

1$\begingroup$ I don't follow. By Cayley's theorem, every group is a set of functions. $\endgroup$ – Qiaochu Yuan Mar 11 '11 at 13:55

$\begingroup$ @Qiaochu: Although every group is isomorphic to a group of functions, the context of the original question ("When students are first learning ...") significantly reduces the appropriateness of working up to isomorphism. In particular, if I'm teaching an introductory algebra class, I'm likely to use examples like the one in the question without immediately mentioning an isomorphic copy consisting of functions. (The isomorphic copy and Cayley's theorem would come up somewhat later.) $\endgroup$ – Andreas Blass Mar 11 '11 at 14:42
Emil's basic observation can be extended. Polynomials are smooth (i.e., infinitely differentiable) functions. What can you say about smooth group laws on the set of real numbers?
In different language: a Lie group is a manifold which comes equipped with group operations which are smooth with respect to the manifold structure. What can you say about Lie groups whose underlying manifold is the set of real numbers with its standard manifold structure?
Answer: they are all isomorphic. In particular, they are all isomorphic to the standard example. That is, there is always a smooth bijection $L$ (with smooth inverse) such that the group operation is given by $x \ast y = L^{1}(L(x) + L(y))$. This is proven in any text which treats Lie groups. So: all examples are "cheats".
Of course, your example is slightly different: it is isomorphic to $(\mathbb{R}  \{0\}, \cdot)$. In fact, all Lie group structures on the punctured line are isomorphic to this one, which is
$$\mathbb{Z}_2 \times \mathbb{R}$$
where the Lie group structure on $\mathbb{R}$ is, of course, given by addition.

1$\begingroup$ I wrote up a handout, for the case of the real line, at a level suitable for someone who just knows calculus: math.uconn.edu/~kconrad/blurbs/grouptheory/relativity.pdf $\endgroup$ – KConrad Mar 11 '11 at 17:21
The only polynomial group law on $\mathbf{R}$ such that the identity element is $0$ is given by the polynomial $P(X,Y)=X+Y$.
Proof. Let $P \in \mathbf{R}[X,Y]$ be the group law, with identity element $0$. For any $y \in \mathbf{R}$, let $P_y$ be the polynomial $P(X,y)$. If $y' \in \mathbf{R}$ is the inverse of $y$ with respect to the group law, then $P_y \circ P_{y'} = X$ so that $\operatorname{deg}(P_y) \cdot \operatorname{deg}(P_{y'}) = 1$. Thus the degree of $P$ with respect to $X$ is 1. Similarly the degree of $P$ with respect to $Y$ is 1. So $P=\alpha X+\beta Y + \gamma XY$. Since $P(X,0)=X$ and $P(0,Y)=Y$ then $\alpha=\beta=1$. Moreover for any $x \in \mathbf{R}$, the function $y \mapsto x+y+\gamma xy$ is a continuous bijection of $\mathbf{R}$, which forces $\gamma=0$.
Note : if the identity element of $P$ is $a \in \mathbf{R}$ instead, then by considering $(X,Y) \mapsto P(X+a,Y+a)a$ we see that necessarily $P(X,Y)=X+Ya$.

1$\begingroup$ Apart from the final $\gamma=0$ step, the same argument works for any infinite subset of $\mathbb R$, which covers the motivating example $x+y+xy$. That is, if $p(x,y)$ defines a group operation on any infinite set of reals, then $p(x,y)=x+ya$ or $p(x,y)=\gamma xy+bx+by+(b^2b)/\gamma$, i.e., a linear shift of $x+y$ or $xy$. $\endgroup$ – Emil Jeřábek Mar 11 '11 at 15:55

1$\begingroup$ It also seems that the same proof works with $\mathbf{R}$ replaced by any infinite field. $\endgroup$ – François Brunault Mar 11 '11 at 17:04

1$\begingroup$ jstor.org/pss/2042946 seems to prove the generalisation to infinite fields and something more. $\endgroup$ – Gian Maria Dall'Ara May 28 '11 at 15:13
All the above observations and much much more can be found in the classic paper:
Hinrichs, L.; Niven, I.; Eynden, C.L.V., Fields defined by polynomials, Pac. J. Math. 14, 537545 (1964). ZBL0122.29103.
Here the authors prove that $f(x,y)$ is a polynomially defined group law on an infinite field $k$ if and only if $f(x,y) = x+y+c$ for some constant $c$ in $k$. They go on to characterize polynomially defined field operations on $R\times R$ as well. Very useful paper from the point of view of this topic especially at a beginning level.  R. Padmanabhan, University of Manitoba.