One name and two books come to mind: Joseph Schillinger, and his two books, The Mathematical Basis of the Arts and The Schillinger System of Musical Composition.
The Mathematical Basis of the Arts is a work that aims at generalizing the concepts present in art pieces, in general, from a geometrical point of view, and how this affects the human perception mechanism. I cannot comment too much about this book because I have read it but haven't really study from it. Schillinger's works are clear, but the (natural) language he uses, and the strange mathematical context/notation make it so that some time and practice are needed to really absorb his ideas. A quick look at the table of contents should wet anyones appetite.
The Schillinger System of Musical compositon is as much a philosopical corpus about aesthetics based on geometrical/psycological/physiological arguments, as a theory of composition. It is the only work that I am aware of that has a philosophical theory of melody. That is, it throughly studies what makes a melody what it is, and not just how to write a melody. The principles that he presents and develops apply to art in general, and not only to music, however, he presents his ideas in the context of musical theory.
I should also say that his is not so much a mathematical theory of art/music/aesthetics as a general theory of aesthetics. It's just that he uses the concepts and language of mathematics to present those general concepts. In doing so, many people come to believe that he is developing a mathematical theory of art/music. He's not. As he points out, he is developing a scientific theory of how we create and perceive art, and thus, effectively, he's developing a formal theory of aesthetics, since beauty is always in the eye of the beholder.
Schillinger's work is from an epoch that believed that human intellect trumped statistical analysis. And that is how it is developed. From a rational derivation from a few basic (universal?) principles, and not from a brute-force approach to find regularities in works of art.