Pros and cons of specializing in an esoteric research area If a mathematician specializes in a popular research area, then there are many job positions available, but at the same time, many competitors who are willing to get such job positions. For an esoteric research area, there are few competitors and job positions. There are very often pros and cons of such research areas.
What are some pros and cons of specializing in esoteric research areas that many people may not know?
Maybe it is a little hard to answer this question in full generality since circumstances vary. Hence, I especially want to listen to examples, personal experiences, and maybe urban legends.
Now, if you are interested, let me tell my personal circumstance to give some context to this question. I am a student from outside of North America who just graduated from my undergraduate institution. Since I decided to study abroad, I applied to several North American universities last December and was admitted to some of them. Now I am wavering between two universities. Denote those universities X and Y. 
Among the specific research areas available at X (resp. Y), I am interested in two of them, say, A and B (resp. C and D). 
I did some searching in those research areas, and I found out that there are many people who are researching C and D and some of them are in my country. However, there are only a few people interested in A and B and none of them are in my country. Based on my search, I think (with a little exaggeration) there are about 3 universities in the world where a graduate student can specialize in A. B is not as esoteric as A, but still, it seems there are not many people working on B. However, I think C and D are quite major research areas in my field of study. 
[In this question, I used 'field of areas' as something in First-level areas of Mathematics Subject Classification and 'specific research areas' as something in Second or Third-level areas of it. ]
At first glance, I prefer X over Y, because I was very interested in A. Also, this is partially (maybe totally) because X is considered more `prestigious’ than Y. However, I’m a little nervous about specializing in esoteric areas such as A and B, because of the number of job positions and this kind of problem.
Anyway, I think it is not a bad idea to ask a question at MO and to listen to the pros and cons of specializing in esoteric fields to make a better decision. Any personal stories or examples will be really helpful. Thanks in advance.
 A: My brother-in-law, a chemist, majored in inorganic biochemistry. Yes, inorganic. Yes, biochemistry. Yes, this subject exists: a handful of chemicals in our body, i.e. hemoglobin, are inorganic. Nevertheless, to my ears "inorganic biochemistry" sounds like an esoteric research area, except that it's in chemistry rather than math.
Unfortunately, the choice of the specialty made his Ph.D. less attractive for potential employers. It's harder to get a grant in the subject NSF didn't hear of. It's also much harder to identify the industry that would need a specialist in such a field. He was searching for employment for a while, and ended up joining US Army, the only organization he found that was researching the biological effects of inorganic chemicals, and got shipped to overseas.
I agree with Gerry Myerson's comment, one should do the things one likes. But, from the narrow viewpoint of post-graduate opportunities after specializing in a common versus an esoteric subject, I would recommend a common one.
A: I think this is an important question, and something that is not talked about often enough. Just like we don't explain enough to undergraduates that the choice of their major has profound consequences for their career opportunities, we don't tell young mathematics researchers enough about the consequences of their choice of research area. (What's the analogue of the well-known joke about the English major serving food in your local restaurant?)
Instead of answering your question, let me give a small analysis of research dynamics, drawn from my own (limited) experience. The below is not supposed to be exhaustive, but I'm hoping to show that there are underlying mechanisms that cause certain areas to be more popular than others.
I'm hoping someone else will post an answer that gets closer to the core of your question, but I hope that this is at least somewhat useful.
What makes interesting research?
This may seem a little random when you're young, but there are actually some underlying mechanisms that are somewhat understandable. (Although admittedly I still haven't figured out a full answer to this question.)
First of all, just like universities and journals, top mathematics research gains some prestige from age. This means that solutions to old conjectures are valuable, and a new theory is appreciated more when it says something about the mathematics that existed before.
But there is a caveat: some areas are considered 'easy' or 'belonging to the past'. There are almost certainly areas that legitimately produce interesting mathematics but are not fashionable because they are considered 'too old'.
Secondly, mathematicians like powerful ideas. If you have some new technique that seems applicable in many situations, this is valued more than an ad hoc argument. If you have a powerful machine that people can use with their eyes closed, that's going to be cited a lot.
Thirdly, we prefer clean theorems. If you have a theorem without too many technical assumptions, then it's much easier to explain and motivate, and much easier for other people to use.
Moreover, connections to other areas are important too: if you prove amazing theorems on a (mathematical) island, then that's not as interesting as when you prove something that relates to other work.
A lot of the above seems driven by the fact that the mathematicians who evaluate your work do so on the basis of their own interests. This seems reasonable given that everyone's own expertise is limited, and their judgement biased towards their own work.
What makes fashionable research?
Areas of mathematics fall in and out of fashion, in part related to the criteria above. For example, if there is a new idea that seems to have potential, this can drive a research group until they feel the idea has been fully exploited. Especially new connections between different areas can spur a lot of activity, because now you have two communities working out these ideas.
When this happens, you can suddenly see a lot of researchers thinking about very similar things, and this is both a blessing and a curse. Indeed, it makes it easier to explain your results and to find collaborators, but it also means you have to keep up with progress all the time and risk getting 'scooped'.
I'm not entirely sure what the mechanisms for topics falling out of fashion are. One thing that can happen is that the main problem gets solved; for example I have the impression that the classification of finite simple groups ended an era of high activity in the area. But this doesn't always happen, because there could be other questions; for example the techniques for proving Fermat's Last Theorem are still very much alive in the Langlands programme.
I imagine it's also possible for a research area to dry out without the main problem being solved, although I haven't been around long enough to know an example offhand.
Note that timing is key: what is fashionable now may not be fashionable in the future, and sometimes researchers forecast the demise of their field. While it's hard to predict the future, it's probably a good idea to listen (critically) when people tell you something like this.
Finally, there are also some political factor involved: if your area is well-represented in the editorial boards of top journals, then you're going to have an easier time publishing in those journals. As Henriksen points out in There are too many B.A.D. mathematicians, this does not always happen for the right reasons.
So what to do next?
Many research groups are driven by a handful of famous questions and a much larger collection of more technical questions. You could ask the research groups you're considering what the main goal in their work is, and then evaluate this by the criteria above to see how easy a time you would have selling your research.
Usually the advice is to talk to some graduate students in the group to hear about their experiences, but in your case it might also be worthwhile to try to approach some postdocs¹ or senior researchers in the area. Another useful metric to look at is job placement: if an advisor has supervised many graduate students who did well on the academic market (short AND long term), this is useful information.
Finally, I should remark that these things matter much more when you're young, because the mathematics job market is highly catered towards the researchers who are lucky enough to have early success. Once you get a tenure-track of tenured job, it becomes easier to switch areas again. This is especially true if you work in an area with connections to other areas, although there are certainly examples of famous researchers switching to completely unrelated fields.

¹Be aware that postdocs are in the most precarious stage of their career, so their answers will be a bit more cynical than those of (young and naive) grad students or (successful) tenure-track or tenured faculty. This can be useful for getting more 'real' advice, but be sure to recognise the context in which it is given.
A: As others have said: I think it's better to focus on the place and the advisor more than the specific field. The place meaning the university, its prestige, its culture etc. as well as the fact you have to live there many years and do very hard work while living there for not huge salary. Some places will be a lot less appealing than others and this matters!
And the advisor in terms of their outlook and culture, the types of jobs their students get, the kind of papers they write (separately from their field I mean are they long? short? technical and computational? or abstract and high-level? etc.). It is commonly said: speak to the grad students to get the scoop on advisors.
A: I think it really depends on your goals. The OP says he's thinking about which grad school to attend. If you're goal is to get a higher degree and then a job in industry, you would be well-advised to do research in something considered "useful." That's not a pre-requisite, and I know a ton of abstract homotopy theorists who now work as data scientists, but it's also fair to say that the transition out of academia is easier if you have some skills you can showcase to companies.
If your goal is to be a professor, I think it's important to realize just how hard it is to get a job these days. The AMS publishes data about the number of PhDs and the number of people who get a tenure track academic position every year. Only about 25% of math phds get a tenure track job. Many of those who get a job in a given year are coming from a postdoc rather than directly out of grad school. When we hire, we get more than 400 applicants for a single job, so even if you do everything right, the competition might still prevent you from getting a tenure track job. Really think about this. A lot of people are jaded when they can't get a job. Don't assume it will be easy or even possible. 
If you want to go to a research university, I think you'd be well advised to pick a less esoteric field, or to connect your research to new and sexy things (e.g., lots of people are getting jobs now in topological data analysis, whereas almost none are getting jobs in chromatic homotopy theory). 
If you are not wedded to the idea of being a professor at a research university, then do not overlook liberal arts colleges. These jobs have a slightly higher teaching load but can still provide a lot of research support, sabbaticals, etc. The essential point for these schools is that teaching is valued more than research, so if you think you might be happy at such a place then it's essential to develop your capabilities as a teacher. Much more important than your area of research is whether or not you can teach statistics or computer science. You get that skill and you will get a job, 100%. You might consider picking up a masters in CS or stats during grad school, as that would help you get a liberal arts job, and would also help if you go to industry. I got a master's degree in CS at the same time that I was getting a PhD in math, and it really did not take a ton of effort. One other point is that liberal arts colleges increasingly want faculty to do research with undergraduate students. Their undergraduate students can be very strong (more like the level of a master's student), but it might be wise to pick a research area that's easier to get into, like graph theory or knot theory, rather than something insanely esoteric. Again, computer science, applied statistics, and data science are very fertile areas for research with undergraduates. 
A: I agree that it is harder to get a job in some subfields than others.  However, aside from some very broad things that are clear to almost everyone (e.g. there are very few people working on point-set topology), I think it is very hard to judge which subfields are "hot" as an undergraduate.  Aside from things that other people have pointed out (like the fact that this is very much a moving target, and that most people you talk to have a very limited and narrow view of the subject as a whole), what "counts" as working in a hot field can be subtle.  For instance, within the subareas I follow, there are certainly hot topics, but there are also topics that are almost entirely dead, and the difference between these topics is only obvious to an insider.  If you don't have appropriate specialized knowledge, the topics look basically the same!
The advice I give to people considering graduate school is to focus less on potential areas (aside from broad things like "algebra" or "geometry"), and focus more on potential advisors.  Go and google the names of your potential advisor's former students (which can be looked up on math genealogy, or on their website if they keep their cv up to date).  See if they're getting the kinds of jobs that you aspire to.  If none of them are, you should take this as an important signal.
Of course, not every student will be equally successful (after all, they're the ones who have to write the thesis!).  What is more, very young advisors might not yet have much of a track record (but it is risky being someone's first couple of students!).
A: Pros:


*

*the research community is small, so it's easier for you to make connections with the top people in the field

*consequently, it should be easier to get good letters from top people and invitations to collaborate

*you are more likely to get invited to speak at a single given conference in your area

*you won't be directly compared to 20 other people your age working on the same area, for things like jobs and grants

*some departments/employers don't care much about you're particular area of work, and will primarily look at things like your publication record and reference letters

*it's easier to stay at the cutting edge of research, and not get scooped, or waste your time proving something you later find out was known 10 years ago
Cons:


*

*the more esoteric your field is, the less well known and influential the "top people" may be outside of your community, making reference letters etc less impactful (in extreme situations, the top people themselves may not be that strong)

*for academic jobs, most research departments will care some about what your field is, and how it fits in with the department interests

*it may be hard convincing editors and referees new results merit publication in top journals

*there is less grant money for your area, and you will need to be really good at selling it to get external grants

*there may not be many conferences in your field, and if you go to other conferences, you may feel "left out"

*there are fewer people for you to talk to about research, and possibly collaborate with (which for me means less fun and excitement)
Thoughts:
There are different levels of how esoteric a field might be, ranging from only a handful of people close to retirement are doing in it to a number of brilliant young people are working in it (maybe it is just "esoteric" because of the high level of technical mastery required for entry), and from being of tangential interest to almost no one versus people waiting for a potential breakthrough that will get coverage in major newspapers around the world.
If you want decent academic options after a PhD, Andy Putman's advice is good: see where people are publishing, sending students, etc to establish a minimum level of having decent prospects.  Certainly there are areas where prospects are just bad.  (Conversely, there are areas where right now prospects seem very good--I'm looking at you data science--but maybe the market will be flooded with data science applicants in 3 years, who knows?  Well, possibly some data scientist already does...)
However, given 2 areas which are not "graveyards", trying to optimize between the above pros and cons based on the relative competitiveness and popularity of them (at least for academic job prospects) seems to me like playing the stocks without serious knowledge of the markets.  You might win by chance, you might lose by chance, but you're trying to make bets based on pretty incomplete knowledge about a situation that is constantly changing.  Thus the best choice is the one that would make you happiest at least in the short-medium term, which is typically some combination of the how appealing the area is to you and finding a successful advisor you're happy with. 
