Mathematical theory of aesthetics The notion of beauty has historically led many mathematicians to fruitful work. Yet, I have yet to find a mathematical text which has attempted to elucidate what exactly makes certain geometric figures aesthetically pleasing and others less so. Naturally, some would mention the properties of elegance, symmetry and surprise but I think these constitute basic ideas and not a well-developed thesis. 
In this spirit, I would like to know whether there are any references to mathematicians who have developed a mathematical theory of aesthetics as well as algorithms(if possible) for discovering aesthetically pleasing mathematical structures. 
To give precise examples of mathematical objects which are generally considered aesthetic, I would include:


*

*Mandelbrot set

*Golden ratio

*Short proofs of seemingly-complex statements(ex. Proofs from the Book)


I think the last example is particularly useful as Jürgen Schmidhuber, a famous computer scientist and AI researcher, has attempted to derive a measure of beauty using Kolmogorov Complexity in his series of articles titled 'Low Complexity Art'. Meanwhile, I find the following research directions initiated by computer scientists particularly fruitful:


*

*Bayesian Surprise attracts Human Attention

*Curiosity and Fine Arts

*Low Complexity Art

*Novelty Search and the Problem with Objectives
Note: From a scientific perspective, researchers on linguistic and cultural evolution such as Pierre Oudeyer have identified phenomena which are both diverse and universal. Diversity is what makes our cultures different and universality enables geographically-isolated cultures to understand one another. In particular, many aesthetics have emerged independently in geographically isolated cultures especially in cultures which developed in similar environments. Basically, I believe that if we take into account what scientists have learned from the fields of cultural 
and linguistic evolution, embodied cognition, and natural selection I think we could find an accurate mathematical basis for aesthetics which would also be scientifically relevant.
 A: A search for aesthetic* in the title at MathSciNet yields 100 hits, e.g., 
MR3751155 Lähdesmäki, Tuuli – Aesthetics of geometry and the problem of representation in monument sculpture. Aesthetics of interdisciplinarity: art and mathematics, 275–290, Birkhäuser/Springer, Cham, 2017. 
MR3751140 Cohen, Mark Daniel – The geometric expansion of the aesthetic sense. Aesthetics of interdisciplinarity: art and mathematics, 29–43, Birkhäuser/Springer, Cham, 2017. 
MR3644156 Pimm, David; Sinclair, Nathalie – Explaining beauty in mathematics: an aesthetic theory of mathematics [book review of MR3156013]. Math. Intelligencer 39 (2017), no. 1, 79–81.
MR3623974 Kao, Yueying; He, Ran; Huang, Kaiqi – Deep aesthetic quality assessment with semantic information. IEEE Trans. Image Process. 26 (2017), no. 3, 1482–1495.
A: One name and two books come to mind:
Joseph Schillinger, and his two books, The Mathematical Basis of the Arts and The Schillinger System of Musical Composition.
The Mathematical Basis of the Arts is a work that aims at generalizing the concepts present in art pieces, in general, from a geometrical point of view, and how this affects the human perception mechanism. I cannot comment too much about this book because I have read it but haven't really study from it. Schillinger's works are clear, but the (natural) language he uses, and the strange mathematical context/notation make it so that some time and practice are needed to really absorb his ideas. A quick look at the table of contents should wet anyones appetite.
The Schillinger System of Musical compositon is as much a philosopical corpus about aesthetics based on geometrical/psycological/physiological arguments, as a theory of composition. It is the only work that I am aware of that has a philosophical theory of melody. That is, it throughly studies what makes a melody what it is, and not just how to write a melody. The principles that he presents and develops apply to art in general, and not only to music, however, he presents his ideas in the context of musical theory.
I should also say that his is not so much a mathematical theory of art/music/aesthetics as a general theory of aesthetics. It's just that he uses the concepts and language of mathematics to present those general concepts. In doing so, many people come to believe that he is developing a mathematical theory of art/music. He's not. As he points out, he is developing a scientific theory of how we create and perceive art, and thus, effectively, he's developing a formal theory of aesthetics, since beauty is always in the eye of the beholder.
Schillinger's work is from an epoch that believed that human intellect trumped statistical analysis. And that is how it is developed. From a rational derivation from a few basic (universal?) principles, and not from a brute-force approach to find regularities in works of art.
A: Being both a professional visual artist and mathematician, I feel obliged to attempt an answer.
There are to me very strong similarities, common mechanisms, overlaps, correspondences, between artistic and scientific aesthetics.
Nonetheless, I usually have trouble explaining that these correspondences go beyond what I consider a more superficial level, which is the level of nice polygons, polyhedra, symmetrical objects,... in other words the typical `mathematical art' that many scientists associate with aesthetically pleasing.
Birkhoff's book is interesting, but to me falls short of addressing the essential complexity of aesthetics, in any discipline. By 'essential complexity' I mean that -in my perhaps not so humble opinion- one cannot approach understanding aesthetics by simplifying to a clearly less complex setting.
Also in mathematics, I have seen people disagree on the beauty of certain proofs or theories. It seems to co-depend on which kind of patterns we can or like to discern...
But I do think that aesthetics guides us in mathematics, and that we all know the gratification of discovering 'beauty'. What is less underscored, is that the discovery of 'non-beauty' can be just as fruitful for furthering our mathematical universes. This I see as a strong parallel with art. Another such parallel is the way in which we associate patterns with `meanings', interpretations, observations,...
We not only discover patterns, but we create them too. Irregularity and asymmetry are as much a part of beauty as regularity or symmetry. Even very imperfect creation has its own aesthetic appeal... and mathematics is a very creative science.
Well, that's my 2 cts worth attempt...I admit Birkhoff gave it a lot more work and attention, and his book is therefore the better enjoyable :-)
A: I think you can do no better than Proofs from THE BOOK,
a collection of mathematical beauties:

Aigner, Martin, and Günter M. Ziegler. Proofs from THE BOOK. Springer, 2014.
(Springer link.)

There is a nice recent interview of Günter in
Quanta Magazine,
where he says:

"We’ve always shied away from trying to define what is a perfect proof. And I think that’s not only shyness, but actually, there is no definition and no uniform criterion. Of course, there are all these components of a beautiful proof. It can’t be too long; it has to be clear; there has to be a special idea; it might connect things that usually one wouldn’t think of as having any connection.
For some theorems, there are different perfect proofs for different types of readers. I mean, what is a proof? A proof, in the end, is something that convinces the reader of things being true. And whether the proof is understandable and beautiful depends not only on the proof but also on the reader: What do you know? What do you like? What do you find obvious?"

A: Chai Wah Wu from IBM’s TJ Watson Research Center has built a machine-learning algorithm using data from OEIS to learn to identify if a sequence will be interesting or not:
[1805.07431] Can machine learning identify interesting mathematics? An exploration using empirically observed laws


Abstract: "We explore the possibility of using machine learning to identify
  interesting mathematical structures by using certain quantities that
  serve as fingerprints. In particular, we extract features from integer
  sequences using two empirical laws: Benford's law and Taylor's law and
  experiment with various classifiers to identify whether a sequence is
  nice, important, multiplicative, easy to compute or related to primes
  or palindromes."

MIT technology review has made an article about this paper:
https://www.technologyreview.com/s/611272/this-algorithm-can-tell-which-number-sequences-a-human-will-find-interesting/
A: What about this paper where the aesthetics of fractal dimension is measured. The peak seem to be (according to the paper) near Hausdorff dimension 1.5.
A: George D Birkhoff, Aesthetic Measure, 1933

An attempt to bring the basic formal side of art within the purview of simple mathematical formula defining aesthetic measure. Contents: the basic formula; polygonal forms; ornaments and tilings; vases; diatonic chords; diatonic harmony; melody; musical quality in poetry; earlier aesthetic theories; art and aesthetics. Over 20 plates and illustrations.

A: I really wouldn't look into Geometry when considering when Aesthetics and Mathematics converge together.
As I've always thought: Mathematically, if there is great order, "beauty" is only a neurological phenomenon of perceiving said order.
