I am thinking about advanced texts similar to Polya's 'How to solve it?'. Quite a few good articles of such a kind are published under Philosophy of Mathematics, but that dwells on a very different domain generally.

12$\begingroup$ Not quite in the style of Polya, but equally insightful, is Hadamard's Psychology of Invention in the Mathematical Field, ia800304.us.archive.org/4/items/eassayonthepsych006281mbp/… $\endgroup$– John StillwellApr 12, 2018 at 4:56

1$\begingroup$ These don't qualify as advanced (except for Villani’s book) and they're certainly not texts, but possibly relevant are: The Mathematical Experience by Phillip J. Davis and Reuben Hersh (originally published in 1981); How Mathematicians Think by William Byers (2007); Loving + Hating Mathematics by Reuben Hersh and Vera JohnSteiner (2011); Birth of a Theorem by Cédric Villani (2015). $\endgroup$– Dave L RenfroApr 12, 2018 at 7:10

13$\begingroup$ I cannot recommend Villani's 'Birth of a theorem'. There are no doubts that Villani is a brilliant mathematician. But the book depicts a guy who is unbearably full of himself. $\endgroup$– Joel AdlerApr 12, 2018 at 10:05

1$\begingroup$ In case there are more people who aren't familiar with that book: see en.wikipedia.org/wiki/How_to_Solve_It to help understanding the OP's question $\endgroup$– Jules LamersApr 12, 2018 at 12:48

1$\begingroup$ Related: mathoverflow.net/q/220052/6518 $\endgroup$– KimballApr 12, 2018 at 14:13
6 Answers
Jacques Hadamard, The Psychology of Invention in the Mathematical Field.
Description (from the Library Journal):
Thoughtful and articulate study of the origin of ideas. Role of the unconscious in invention; the medium of ideas — do they come to mind in words? in pictures? in mathematical terms? Much more. "It is essential for the mathematician, and the layman will find it good reading."
There is something interesting in page 118 on the Riemann Hypothesis:
At the death of Riemann, a note was found among his papers, saying 'These properties of $\zeta(s)$ (the function in question) are deduced from an expression of it which, however, I did not succeed in simplifying enough to publish it." We still have not the slightest idea of what the expression could be.

$\begingroup$ Do you know what are the properties he was talking about? $\endgroup$– lcvMar 6, 2020 at 17:35
This question has, as already underlined, many different dimensions. It is not only a matter of psychology but also of epistemology, philosophy, history and sociology. However, your question and Polya's example suggest a more mathematicallyintended answer than many general reference about proofs and discoveries. I try to give three answer in three very different directions, and I believe most of the relevant references can be found in the direction of what is known as "sociology of scientific knowledge (SSK)".
A mathematician's answer. I wonder why this is not yet in the answers and comments, but without further comments (in French, I cannot imagine it has not been translated, but don't find it quickly in English):
A. Grothendieck, Récoltes et semailles. Réflexions et témoignage sur un passé de mathématicien.
A teacher's answer. I can advise the epistemologic
W. Byers, How Mathematicians Think, Princeton University Press, 2007
The book is written by a professor of mathematics who delves into the creativity involved in mathematical research, and not only formalism or scholarship (in the sense having a large culture). Though, it is sometimes quite personal and written with basic examples, for it is also intended for a more general audience.
A sociologist's answer. Also, a more sociological work on the professional behavior and conceptions of mathematicians, intended to supply a synthesis of many traits of mathematicians beliefs, is (originally in French, I do not know whether or not it has been translated)
B. Zarca, [The Universe of Mathematicians. The professional ethos of the most rigorous scientific], Presses Universitaires de Rennes, 2012
Unlike the previous one, it emphasizes more the interrelation between proofs, community, tools and strategies. Also, a lot is dedicated to the judgement of mathematicians about what they do (interest, belief, esthetic, pleasure, challenge, elitism, pride), what may match more your original question and its psychological aspect (even more than the more epistemic examples given by mathematician themselves).

5$\begingroup$ ncatlab.org/nlab/show/R%C3%A9coltes+et+semailles contains a link to a partial translation of Récoltes et semailles to English $\endgroup$ Apr 12, 2018 at 17:48
I think there are probably a few mathematicians that have deal with this problem better than the ones that have been simultaneously active in a completely different area, and from the ones that I know, I would suggest you to read GianCarlo Rota.
In particular, I would suggest you the book Indiscrete Thoughts, and chapter 9 ( The reductionist concept of the mind) from his essay The Pernicious Influence of Mathematics upon Philosophy.
In this chapter he asks: What does a mathematician do when trying to work on a mathematical problem? He starts citing Polya and saying that the most important step is look at other attemps, then he tells about mathematicians looking for the story of problems in order to solve them...A real story or an ideal reconstructed story a gifted mathematician may create.
He explains that the real nature of a mathematical problem is something that mathematicians are daily discovering (not something fixed or completely known) by a ''handinhand'' passing of solutions of mathematical problems, and he associated this to an historic process.
He compares the way the mathematical thinking differ from the philosophers: hard thinking and blank mind is not the way how a mathematician thinks.
He concludes saying that ''the process of the working of the mind, which may be of interest to physicians but is of no interest to mathematicians, is confused with the progress of thought that is required in the solution of any problem''.
I would also suggest you to try to extract your own conclusions from the bibliographies, quotes and behaviour of your favourite mathematicians. You may start with the list of Alex Bellos: Pythagoras, Hypatia, Cardano, Euler, Gauss, Cantor, Erdős, Conway, Perelman and Tao.
If you are thinking about about advanced texts similar to Polya's 'How to solve it?', I would suggest you Solving Mathematical Problems by Terence Tao and his article https://terrytao.wordpress.com/careeradvice/solvingmathematicalproblems/.
If you are looking to read divulgation books related to the psychology of mathematical research, I would also suggest you: Fermat's Last Theorem: The Story Of A Riddle That Confounded The World's Greatest Minds For 358 Years by Simon Singh, Birth of a Theorem: A Mathematical Adventure by Cédric Villani, and to look at books/articles by Alex Bellos.
Finally, some well known quotes from famous mathematicians that are related to the psychology of mathematical research:
Roger Penrose mathematical thinking:
"My own way of thinking is to ponder long and, I hope, deeply on problems and for a long time ... and I never really let them go."
Andrew Wiles explanation of how is his experience working in mathematics.
“Perhaps I could best describe my experience of doing mathematics in terms of entering a dark mansion. You go into the first room and it's dark, completely dark. You stumble around, bumping into the furniture. Gradually, you learn where each piece of furniture is. And finally, after six months or so, you find the light switch and turn it on. Suddenly, it's all illuminated and you can see exactly where you were. Then you enter the next dark room...”
“I carried this problem around in my head basically the whole time. I would wake up with it first thing in the morning, I would be thinking about it all day, and I would be thinking about it when I went to sleep. Without distraction I would have the same thing going round and round in my mind.”
Alan Turing thought about what is mathematical reasoning, and how intense it is.
“Mathematical reasoning may be regarded rather schematically as the exercise of a combination of two facilities, which we may call intuition and ingenuity.”
“I have such a stressful job that the only way I can get it out of my mind is by running hard.”
John Horton Conway comparison of doing maths with playing games.
You get surreal numbers by playing games. I used to feel guilty in Cambridge that I spent all day playing games, while I was supposed to be doing mathematics. Then, when I discovered surreal numbers, I realized that playing games IS math.
And the last quote, the beautifulness of the number by Paul Erdos:
Why are numbers beautiful? It’s like asking why is Beethoven’s Ninth Symphony beautiful. If you don’t see why, someone can’t tell you. I know numbers are beautiful. If they aren’t beautiful, nothing is.
That serves as a motivation to ''The experience of mathematical beauty and its neural correlates'' Front. Hum. Neurosci., 13 February 2014, a study that relates mathematical beauty to artistic beauty, as producing stimulations in the same part of the emotional brain.
Henri Poincaré, “Science and Hypothesis”. This is maybe a bit more philosophy then psychology, but I still think it is relevant.
“Poincaré had philosophical views opposite to those of Bertrand Russell and Gottlob Frege, who believed that mathematics was a branch of logic. Poincaré strongly disagreed, claiming that intuition was the life of mathematics. Poincaré gives an interesting point of view in his book Science and Hypothesis:
‘For a superficial observer, scientific truth is beyond the possibility of doubt; the logic of science is infallible, and if the scientists are sometimes mistaken, this is only from their mistaking its rule.’ ”
(Quoted from wikipedia:Henri Poincaré)
“Aimed at a nonspecialist readership, it deals with mathematics, space, physics and nature. It puts forward the theses that absolute truth in science is unattainable, and that many commonly held beliefs of scientists are held as convenient conventions rather than because they are more valid than the alternatives.
In this book, Poincaré describes open scientific questions regarding the photoelectric effect, Brownian motion, and the relativity of physical laws in space. Reading this book inspired Albert Einstein's subsequent Annus Mirabilis papers published in 1905.”
(Quoted from wikipedia:Science and Hypothesis)
Fernando Zalamea’s Synthetic Philosophy of Contemporary Mathematics has a chapter called “phenomenology of mathematical creativity” which I think does an excellent job of tackling that topic (a very psychologically charged aspect of mathematical research) since Zalamea is also a talented art critic/theorist (so has a good understanding of creativity at large) and is much more deeply enmeshed in actual mathematical practice than lots of philosophers are*.
*fortunately this is becoming more and more common with eg the existence of the association for the philsosophy of mathematical practice
I think the book "Advanced Mathematical Thinking" By "David Tall" is useful for your direction. The book begins with the title "psychology of advanced mathemacal thinking" and goes on by the topic "The nature of advanced mathematical thinking" and cognitive theory. As mathematical research needs mathematical thinking, I think this book is interesting for you.