This is really an add-on to David Corfield's answer.

Since David mentions groups and groupoids, I will mention that Ronnie Brown ([http://groupoids.org.uk/hdaweb2.htm][1])  considers some of the possible criteria as follows:

Tests for a theory which is successful in a mathematical and scientific rather than sociological sense could be the following. A successful theory would be expected to yield.  He wanted to evaluate some new concepts and proposed the following advantages.

 - a range of new algebraic structures, with new applications and new results in traditional areas;
 - new viewpoints on classical material;

 - better understanding, from a higher dimensional viewpoint, of some phenomena in group theory;

 - new computations with these objects, and hence also in the areas in which they apply;

 - new algebraic understanding of the structure of certain geometric situations;

 - a stimulus to new ideas in related areas;

 - a range of unexplored ideas and potential applications;

 - the solution of some classical famous problems. 

I would suggest that this list (albeit incomplete as Ronnie suggests) applies to algebraic situations as well as his higher dimensional group theory context and that, suitably interpreted for other contexts, they can provide some very partial answer to the question. 

The second question is perhaps best answered by saying that 'established' mathematicians are expected to have some sort of 'gut' feeling about the importance of a question or area. Sometimes they just have blind prejudice however. One task of a research supervisor 'should' be to train a PG student towards getting that intuition, but not to hand on the prejudices. 

At a pragmatic level a debutant mathematician needs to get work published and noticed and that is easier in established areas (or near established areas).


  [1]: http://groupoids.org.uk/hdaweb2.htm