I have to disagree strongly with Gerald Edgar.

The controversial book The Topos of Music by Guerino Mazzola could constitute a very serious attempt to answer your question. This is not a book that would be accessible to a typical musician, or even a typical expert in music theory - it is definitely a mathematics book about music, taken seriously by some eminent mathematicians as claimed here

http://en.wikipedia.org/wiki/Guerino_Mazzola

and probably in need a refinement.

The question "what is music?" might entertain a musician, a philosopher, an anthropologist, a sociologist or, as Edgar suggests, a psychologist, and they would have something (different) to say. The more precise question here might be "what mathematical structure provides an adequate model for anything someone might create and call a musical composition?" The problems here art that musical compositions have multiple realizations; realizations of a score are constrained by various axioms; and composer specify music compositions by choice with greater or lessor degrees of determinacy. Mazzola seeks a unified model that can represent the whole range of what modern composers (and not just of Western art music) actually do.