There was time when the paradigm of self-similarity was applied to pretty much everything, including music. This article is one of the attempts made in this direction (and mentions some other ones):

http://www-acad.sheridanc.on.ca/~degazio/pdf/Musical%20Aspects.pdf

It includes e.g. a reference to "the study of music as a scaling (1/f) noise".

One can find a lot of "fractal music" on the web, but the connection with mathematics is usually very remote.