When going off on a tangent from your regular area, where, presumably, you have such mastery of all cutting-edge research from your routine reading that you hardly need to do any extra (if this is false, please correct me), how much do you try to familiarize yourself with that area before beginning to directly attack your problem? Do you read just a few canonical papers and surveys, look thoroughly over a dozen and glance at a couple dozen more, or do enough to write a whole survey article of your own?
When you know the definitions, of the elements of your problem, no doubt, start attacking the problem. Ever if you still don't know the definitions for the most general form of the problem and only for a simplified version of it the answer is the same. No better sense of what the problem is about than by putting your own hand on it. Even if it only serves to get to some conclusion that you could have easily read somewhere. It could even happen that you solve the problem by putting together two or three things that you read but more important in research than solving the problem is understanding it. Because after solving it, you need to find new problems to continue. You don't understand anything better than those that you do yourself. The above doesn't mean don't read. It means don't wait a second before start doing all you can yourself.
When I attacked the problem of finding a quantum factoring algorithm, I had read four or five papers on the subject, which constituted nearly all the literature on quantum algorithms at the time. However, there were lots of other relevant papers that I didn't even know about in the field which would later be called quantum information theory, and I didn't feel compelled to do a literature search to find them.
This is one extreme. If you try to go this route, you may very likely miss some important techniques that are commonly used in the new field, so I would actually recommend substantial reading in the new field. I had to do that when I started working on quantum information theory.
It would make a lot of sense if you worked on your problem while you did this substantial reading (even though you're liable to go in the wrong direction), because that will be a good guide for choosing which papers you should read. It also helps if you have a colleague in the new area you can talk to or collaborate with. So, as Deane says in the comments, "It depends."
I try to get myself a coauthor who knows that area.
I think reading while you are already working on the problem could be helpful in both directions. On the one hand, while reading you will be more alert to stuff that might be useful for solving the problem, so you will have better chance of paying attention to it. On the other hand, if you already have a problem in mind and you are trying to see how to use the stuff that you are learning to solve the problem, then your learning process is more active and your understanding of the material will be better (or at least some aspects of it).