Is amateur research in mathematics viable? After a long reflection, I've decided I won't go to graduate school and do a thesis, among other things. I personally can't cope with the pressure and uncertainty of an academic job.
I will therefore move towards a master's degree in engineering and probably work in industry. However, I'm still passionate about math, and will continue to attend seminars, conferences, and work with people in my heart field (ie: algebraic geometry and number theory).
My question is: Is it viable?  Won't I be "ostracized from the math community"? Eventually, could I still publish work?
 A: There where a few famous 'amateurs' in the past: Fermat worked as lawyer (but at his time there were of course (almost) no permanent posititions).
Ramanujan had no formal instruction.
(The list is certainly much longer.)
I think the following tipps (with some overlap over previous answers) are useful:

*

*Chose your domain well (not something too technical, but still interesting).


*Use standard terminology and notations.


*Write in Tex/Latex (mathematicians get very suspicious over proofs of the Riemann hypothesis written up in Word)


*Do not despair if you rediscover some results which are already known : This proves that you are not so bad (this happens also to professional mathematicians and it is in any way much better than to have new results which turn out to be wrong, which happens also quite often but 'errare humanum est' and analyzing mistakes is excellent for learning).


*Good luck (I have always the impression that I
find new results mostly by being lucky).
A: This is possible. I have at least two friends who studied mathematics (in the graduate school), did not defend their PhD, and found some jobs not related to mathematics. Still they do research, and publish papers from time to time.
Probably the most famous modern mathematician who never studied mathematics on a graduate level was Marjorie Rice.
She made an important contribution.
The main problem on my point of view is not being "ostracized by math community" but the problem of lack of time for concentration on mathematics. Those two of my friends who started publishing are both retired, one had a career in business another in computer programming. Several of my other friends, who did have a PhD had to switch to other activities simply because they could not find jobs in
mathematics. Many of them were intending to continue their math research "in free time". But the problem is that there is usually no free time  if you do another job. Mathematics requires a high degree of concentration.
A: Even most low-grade office jobs outside of academia these days are surprisingly high pressure and stressful. Your performance is constantly monitored and analysed. I would be surprised if jobs in industry were less high pressure.
If you want to continue mathematics on the side and are certain you can get results, it might be easier to get a basic office job where not so much is expected of you. Depending on how relaxed your line manager is, you might be able to get away with just sending a few emails and filling out some forms and then essentially have the rest of the day to think.
A: Although I studied Economics (about 15 years ago), never attending any math lecture at the university, I have finally published some original research papers in peer-review journals (about number theory and graph theory, mainly) after some hard work as an autodidact (studying online without any external help), focusing myself on just a couple of subfields and then putting most of the effort on a few very specific topics (e.g., number theory ⟶ hyperoperators, graph theory ⟶ optimal path covering, and so forth).
A pair of things I wish I would have learned before are as follows:

*

*Read related papers before starting to develop your own results.


*Never submit papers in a rush, just take your time to review them carefully.
A: Anything is possible, but what you propose is very very very hard. You can certainly learn new stuff for fun, but producing research may be challenging. Amongst many, a big factor could be the lack of collaboration. When you are inside a circle, you have access to experts. Without quick access to an expert, even a simple conceptual glitch or a tiny knowledge gap may hold you back for an inordinately long time (personal experience).
A: As someone who has gone through a in-some-ways similar experience, I don't think you would be ostracized, and yes, you absolutely could still publish work. You might need people to endorse your work, though, for example if you lack an academic affiliation but want to publish on the arXiv.
Generally speaking, people will be interested in your theorems, and maybe especially people who already know you as a mathematician. Stay in touch with your mathematical friends and contacts.
A: A number of people I know and work with, including myself, are math PhDs currently working in industry after having tried to pursue a career in academia. Several of them, including myself, are trying to stay active in research. Some observations:

*

*It is possible, but very difficult. Primary obstruction is time - what you could have read/understood/written in a day or two will take weeks, perhaps months. It is even more difficult when/if you have children. Some of my colleagues completely stopped research for a few years when the children were young.


*On the positive side, you will be forced to be more disciplined. I think I am more productive (per unit time) now than any time in the past.


*It is natural, and actually more sustainable, that your day-to-day work will have an effect on your interests. Several of the most productive people I know now publish in financial math (their area of work) even though their thesis was in number theory or some other unrelated field. I am also working on a statistical problem which came up during a project at work and I found interesting.
A: While I agree with others that it is possible to pursue mathematical research as an amateur, and I don't think you'll be "ostracized," I do think that there are some potential sociological obstacles that you should be aware of.
If you want to publish a paper, then you have to write your paper in a way that editors and referees will take seriously.  Editors have been known to dismiss papers unfairly for all kinds of reasons, ranging from a missing citation to the use of Microsoft Word instead of $\mathrm\LaTeX$.  As I recall, even Yitang Zhang's extraordinary paper on bounded gaps came perilously close to being ignored, but fortunately his writeup was picture-perfect. I am not sure how much experience you have with writing mathematical papers; if you don't have a lot of experience, then I would recommend that you seek out opportunities to gain such experience and obtain constructive feedback on how you can improve.  One possibility would be to submit papers to something like Mathematics Magazine, which does not require original research but does demand high-quality writing.
If you can form some kind of personal relationship with one or more professional mathematicians, that will also be a big help.  Even posting an arXiv preprint requires an endorsement.  The mathematician may also be able to give you good advice about relevant literature, where to submit your paper, what corrections need to be made to your first draft, etc.
A: I hope I may add my personal, anecdotal experience - which I think mostly backs up the answers already given. I did a PhD in an area of mathematics not directly related to my favorite subject, character theory, and then saw no opportunity to remain in academia and trained as an accountant, later working in quantitative finance. But the desire to do character theory crept back.
It happens that in character theory there are some generous professional mathematicians, who responded kindly when I badgered them with my work, and helped me publish a few papers over the years. It takes longer to get them published, and perhaps sadly they are not read as much - that is where the real satisfaction comes - but I've had nothing rejected that did not, on reflection, deserve it. I don't know if the warmth I was shown is subject - specific but I doubt it; of all people, mathematicians somehow don't take themselves too seriously.
On the other hand - and I feel this painfully when I occasionally attend conferences - modern mathematics is intensely collaborative. There is a sense of excitement, urgency almost, in working to solve the big problems. There's a lot of strategizing, and everyone knows everyone with no formality. Sleeves are rolled up, as it were. I think it is particularly hard as any kind of outsider, with the best will in the world, to feel part of that.
I don't have the personality (or the intellect) to feel I can play a serious role in that arena, and I do think it is more difficult to produce ideas without bouncing them off colleagues. As such, and not being a lone genius, I'm a picker-up of odds and ends. But that has been a big satisfaction for me nevertheless. At least, there is a lot of pretty mathematics still undiscovered in the less deep parts of every subject, I like to think.
A: I think the real viability question here is whether you can maintain your passion for math outside of an academic context. It's certainly possible, but you really have to be motivated to make this work.
If you're doing good work, your other concerns aren't serious obstacles.
A: After I left academia, I kept up with doing academic work. What works well for me is to work on a wide range of subjects. So, while my background is in mathematical physics, I use the freedom I have by virtue of not being in academia, to work on whatever I want to work on in my free time. I'm not subject to any pressure to get  anything I'm working on finished. It's not a problem if I have no results for several years. This allows me to reach results that are not easily reachable by regular academics.
A: Something that I don't see mentioned in other answers is that a number of research engineers in academia are making substantial contributions to mathematics. Simply put, many excellent applied mathematicians choose to be employed as engineers in academia. This may be the most natural route for someone interested in both engineering and math research. Of course, this does not remove "the pressure and uncertainty of an academic job", which you are (quite correctly) worried about. In any case doing a master at a top engineering school will get you exposed to some of these people, and you can seek their advice.
A: The "tl;dr" answers to your questions:

*

*Not really

*Quite possibly

*Probably not

What you have at the moment is just an aspiration. You really love algebraic geometry and number theory. But you do not know for sure whether or not you're capable of undertaking research in either field. Attending graduate school would tell you whether or not your aspirations are realistic.  In the process, you'd be filling in the necessary background in the field, getting a view of the relevant literature, getting insight in ways of tackling research problems and overcoming obstacles, and practicing writing up your results for publication.  It's difficult to learn all that on your own.
I should add that the mere fact that you could enrol in graduate school doesn't necessarily mean that you'd be successful. I don't know the successful PhD completion rates in your country, but they're very low where I live (Australia).  Obviously many intelligent, well qualified people start out with high hopes and aspirations just like yours, only to exit early. It could be that they find there's a mismatch between their vision of doing research and the reality.
Several answers here have offered examples of people successfully undertaking independent research. Unfortunately, these examples indicate nothing whatsoever about your chances as an individual: survivorship bias spoils the party yet again.
As for being "ostracized from the math community", the reality is that "independent researcher" is a very broad term. It includes mathematicians with a proven publication record, who haven't been able to secure a university position.  But it also includes many misguided and even insane people. So you'd need to work hard to maintain and extend any connections you may have with the mathematics community.
Similarly, getting your work published could be problematic. When it comes to examining and endorsing unsolicited papers from an unknown source, I strongly suspect that many mathematicians just don't have time to filter out the trash and investigate the few good items.
A key point that hasn't been mentioned so far is the problem of access to mathematical research and libraries.  The widespread shift from paper to electronic formats restricts this access. Maybe you could still visit your local university library and browse, but much of the material can only be obtained online.  If you're not a university employee or postgraduate student, obtaining access may be difficult or impossible.
ETA: off topic for this forum, but you may well find that the pressure and uncertainty of an engineering career far exceeds that of an academic career.
A: I have this image of science as an expanding sphere, with ever more unsolved problems, but at an ever larger distance from the center. In reality it is probably some kind of curved high dimensional manifold. Maybe there are some nooks and crannies you can find as an amateur, I couldn't really say. But if you enjoy learning there isn't really anything stopping you and who knows what you might find?
