There won't be any `right' answer here because there are many different ways that one can come across groups in studying various algebra problems, but here are a few examples:

Maybe the most famous exotic case is when you classify the definite inner product algebras over $\mathbb{R}$:  Such an algebra $\mathbb{A}$ is an $\mathbb{R}$-algebra with unit $1$ endowed with a positive definite inner product $\cdot$ that satisfies the multiplicative identity $xy\cdot xy = (x\cdot x) (y\cdot y)$.  It's a classical theorem (due, I think, to Dickson) that the $\mathbb{R}$-dimension of such an algebra is one of $1$, $2$, $4$, or $8$, and the corresponding algebras are $\mathbb{R}$, $\mathbb{C}$, $\mathbb{H}$ (i.e., Hamilton's quaternions), or $\mathbb{O}$ (i.e., the octonions of Graves and Cayley).  The group of automorphisms of $\mathbb{O}$ is the exceptional group $G_2$ (compact, connected, of dimension $14$).

$G_2$ and its noncompact dual, $G_2^\ast$ also arise as the stabilizer groups of `generic' alternating $3$-forms in dimension $7$.

Similarly, Spin(7) as a subgroup of $GL(8,\mathbb{R})$ turns up as a stabilizer when you go to classify the alternating $4$-forms in dimension $8$.

The exceptional group $F_4$ arises as the autormorphisms of the `exceptional' Jordan algebra of dimension $27$.  ('Exceptional' in this context means that its the one irreducible example that doesn't fit into one of the standard series.)

Depending on your tastes and background, there are many more examples of this nature that might appeal to you.