It is no not what you want, but may be worth mentioning, but there . There is a huge branch of abstract harmonic analysis on (abelian) locally compact groups, which generalizes Fourier transformation on reals and circle. The main point about sin and cos (or rather complex exponent $e^{i n x}$) is that it is a character (continuous homomorphism from a group to a circle) and it is not hard to see that those are the only characters of the circle. That what makes Fourier transform so powerful. If you generalize it along the direction which drops characters, you'll probably get a much weaker theory.
It is no what you want, but may be worth mentioning, but there is a huge branch of abstract harmonic analysis on (abelian) locally compact groups, which generalizes Fourier transformation on reals and circle. The main point about sin and cos (or rather complex exponent $e^{i n x}$) is that it is a character (continuous homomorphism from a group to a circle) and it is not hard to see that those are the only characters of the circle. That what makes Fourier transform so powerful. If you generalize it along the direction which drops characters, you'll probably get a much weaker theory.