I think this is a difficult and significant question, and I have appreciated the answers to date. I am hoping to learn more about this topic from further answers, since I have struggled with this challenge all my career. I think it is widely felt that great problem finders are more rare than great problem solvers. Still I am motivated to try to answer it myself although I don’t feel well qualified to do so. And I hope I may learn something by thinking about it. I suggest a naive preliminary question: is one trying to solve the deepest possible problem, or to have as much fun as possible? Some of us are, to recall David Riesman, “inner directed”, and want to emerge from a private space with a solution of the Riemann hypothesis, and some of us are “other directed”, and just want to show up at a meeting with our advisor with a solution of his/her favorite problem.
Also, there is the question of how should a research problem be ideally chosen, say by a master, versus the question of how should the modestly gifted among us actually proceed, given our limitations. Thus even if one does hope to enter into research on a significant problem as soon as possible after tools begin to exist for its attack, only a select group of people may realize when this occurs.
So for this it is beneficial to maintain contact with the words and writings of those leaders who have command of the field one works in. It also helps if they are conversant with pregnant but little known literature, such as the papers of K. Petri, shown to me in the late 1960’s by my first advisor Alan Mayer, or the book of Wirtinger on Theta functions revealed to me by my second advisor C.H. Clemens.
After being launched by these generous gifts, an opportunity occurred again during a research postdoctorate at Harvard, privileged to be among the giants: Mumford, Griffiths, Hironaka, Mazur, Kazhdan, Bott, Zariski, a fabulous group of students: Bob Friedman, Joe Harris, Ron Donagi, Dave Morrison, Ziv Ran, Rick Miranda,.... and the many other stars who came there - Igusa, Fay, Teissier, Freitag, Tai, Siu, Ramanan, .....
To take advantage of this opportunity, I moved my family to Cambridge and lived on an NSF stipend so small I sold my car the first year for food, and had to decline the second year entirely. So you could say that to pursue excellent current problems from a privileged perspective as a young researcher, I embraced temporary poverty. The benefit was a seat at the theater to which the most active players in my field came to present their latest work. At this point it was very stimulating to try to answer any question whose answer was interesting to one of my mentors but unknown to them.
Those comments are from/for someone of average ability trying to compete for early progress on problems that are of wide spread interest and that may bring notoriety. Fortunately, as much or more satisfaction is found by working on problems that just appeal to our imagination, and that match our own expertise. At this later stage we are moving away from dependence on experts, to instruct us and supply us with topics and ideas, and are beginning to follow our own interests.
So here one begins to acquire some expertise oneself, from study and independent work. It then begins to be ones own responsibility to maintain up to date awareness of the progress of others and to try to apply it to questions that appear of interest. It seems crucial here to attend talks by the best workers and to read their works. At this point one reaches the mystical stage of being able to predict what the answer to a question will likely be, before one has solved it. One may even attempt to compete with recognized experts on the same problems. Success however will depend on more than good intuition, but also on mastery of technical tools to complete the work.
Some very strong individuals work more privately and still attack more public problems. I recall William Fulton saying he wanted to try to understand Schubert’s work on enumerative geometry, so he started reading it and filling details, but I don’t recall how much the fact it was a Hilbert problem influenced him, if at all. My colleague Bob Rumely was attracted to a Hilbert problem on finding procedures for solving integer equations, and tweaked it brilliantly to arithmetic integers. So another problem - finding technique is to take a well known one, solved or unsolved, and modify it intelligently.
The third stage it seems to me, is having such a wide awareness of the state and likely development of a field or area, that one sees likely problems on every hand and invites new talent to work on them. If one succeeds here, one may create a team and an environment of creative research that feeds on itself, and all the players may learn to add to the palette of interesting problems.
By the way, I would very much like to read accounts from some of the participants here of how they found a few of their favorite research problems. Going out on a limb here, I suspect a strong element of randomness will be noted.