Having said this, I do rather like the following example: Kac-Moody algebras could be considered 'idle' generalizations of finite-dimensional simple Lie algebras. One considers the construction of simple Lie algebras by generators and relations starting from a Cartan matrix. When a positive definiteness condition is dropped from the matrix, one arrives at general Kac-Moody algebras. I'm far from knowledgeable on these things, but I have the impression that the initial definition by Kac and Moody in 1968 really was somewhat just for the sake of it. Perhaps indeed, the main (implicit) justification was that the usual Lie algebras were such successful creatures. Other contributors here can describe with far more fluency than I just how dramatically the situation changed afterwards, accelerating especially in the 80's, as a consequence of the interaction with conformal field theory and string theory. But many of the real experts here seem to be rather young and perhaps regard vertex operator algebras and the like as being just so much bread and butter. However, when I started graduate school in the 1980's, this story of Kac-Moody algebras was still something of a marvel. There must be at least a few other cases involving a rise of comparable magnitude.
Maybe I will add one more personal comment, in case it sheds some darkness on the question. I switched between several supervisors while working towards my Ph.D. The longest I stayed was with Igor Frenkel, a well-known expert on many structures of the Kac-Moody type. I received several personal tutorials on vertex operator algebras, where Frenkel expressed his strong belief that these were really fundamental structures, 'certainly more so than, say, Jordan algebras.' I stubbornly refused to share his faith, foolishly, as it turns out (so far).
Added again:
In view of Andrew L.'s question I thought I'd add a few more clarifying remarks.
I explained in the comment below what I meant with the story about vertex operator algebras. Meanwhile, I can't genuinely regret the decision not to work on them because I quite like the mathematics I do now, at least in my own small way. So I think what I had in mind was just the platitude that most decisions in mathematics, like those of life in general, are mixed: you might gain some things and lose others.
To return briefly to the original question, maybe I do have some practical remarks to add. It's obvious stuff, but no one seems to have written it so far on this page. Of course, I'm not in a position to give anyone advice, and your question didn't really ask for it, so you should read this with the usual reservations. (I feel, however, that what I write is an answer to the original question, in some way.)
If you have a strong feeling about a structure or an idea, of course keep thinking about it. But it may take a long time for your ideas to mature, so keep other things going as well, enough to build up a decent publication list. The part of work that belongs to quotidian maintenance is part of the trade, and probably a helpful routine for most people. If you go about it sensibly, it's really not that hard either. As for the truly original idea, I suspect it will be of interest to many people at some point, if you keep at it long enough. Maybe the real difference between starting mathematicians and established ones is the length of time they can afford to invest in a strange idea before feeling like they're running out of money. But by keeping a suitably interesting business going on the side, even a young person can afford to dream. Again, I suppose all this is obvious to you and many other people. But it still is easy to forget in the helter-skelter of life.
By the way, I object a bit to how several people have described this question of community interest as a two-state affair. Obviously, there are many different degrees of interest, even in the work of very famous people.