Pasha mentions presentations of groups; in this connection, I think there are few better examples than the Coxeter groups, with such beauties as $\langle a, b : a^2 = b^2 = 1, (ab)^3 = 1\rangle \cong S_3$.
In keeping with the theme of non-examples, I remember being puzzled by the notions of reductivity and quasi-split-ness until someone told me that a Borel subgroup of an algebraic group (like the group of invertible, upper-triangular $n \times n$ matrices) isn't the former, and the multiplicative group of a skew field (considered as an algebraic group over its centre) isn't the latter.
