I am totally new to academia so I am really not sure how mathematicians works together, can more experienced mathematicians here shed some light on how you find coauthors? I guess one way to do this is to attend conference. But if one doesn't have the chance to do so, are there other options?
Alright, I'll try to outline several ways of how mathematicians find co-authors (since I'm still quite young, I lack the experience to judge how things might change later during a career, but given that you're new to academia, I hope that what follows fits your situation):
I'll try to write my answer as a kind of classification scheme.
Preliminary remark concerning terminology:
Actually, mathematicians do not find co-authors - they find collaborators. By this I mean that it is probably not a good idea to approach people in order to "write a paper with them", but rather in order to "discuss and work with them on an interesting problem or idea." If this work turns out to be fruitful (and novel), you can then turn your results into a paper that you co-author together.
When you're new to academia, you are most likely doing a PhD (or something related), so you will have an advisor. In some (not in all) cases your advisor is the most natural choice to discuss your ideas, seek for explanations, and so on. Depending on how large your advisor's share of your research is, you might write a paper alone or together with your advisor.
How to start a collaboration with your adviser:
Please, be advised that there are very different types of advisors who work and behave quite differently. Some might push you quite often und actively urge you to work with them, others are much more reserved and will let you do your thing unless you actively go for a collaboration with them.
Make sure that you find out and keep in mind what type of advisor yours is.
Other people from your institution.
Another natural choice is to work with other people from your institution. This can vary in several ways:
You might work with people at your level, with people who are a bit more advanced in their career than you are, or with people who are much more experienced than you. When you are completely new to academia, it can be helpful to work with people who have a bit more experience than you (say, more advanced PhD students, or postdocs). They can offer you a bit of advice and guidance (in addition to your advisor).
You might work with people who are very close to your field or research project, or you might discuss with people who have specialized in other things, and when you find a common question of interest, you can combine your knowledge and ideas from two different fields to attack the question. Note that this can range from people working in fields nearby to people who do things that are, at first glance, very different from yours (but I would suspect that the "younger" you are, the better it is if you work with people with similar background to yours).
How to start a collaboration with people from your institution:
First rule: Meet people, and discuss with them - at lunch, at tea, whenever. Do not sit alone in your office or at home all the time (well, Corona won't last forever...). Do not think "I'll discuss with XYZ now, maybe we can work on a project together", but rather think "I have a question in mind in which XYZ might be interested; let's see what she/he thinks about it."
Of course, details depend on which fields people are working in. Is there a PhD student or a postdoc with a field of research that is similar to yours? Go and discuss with them as often as you can! Ask them for their expertise, ask them questions you are thinking about (not necessarily research questions; also things which are probably known, but not to you, yet). Ask them what they are thinking about. Is there a postdoc in a field that is related, but not quite them same as yours? Ask them which problems they find interesting, and why. Ask them if they know how their work relates to your field, and if they have done similar things as you do.
After discussing a topic that you found interesting, re-think what you discussed, look things up, keep coming back to your colleagues with new information, new insights or new questions. Similarly, if somebody keeps coming back to you, and you find their questions or insights interesting, take the opportunity to discuss even more with them. After all, collaborations are not planned or directed - they grow.
One additional remark:
Sometimes collaborations which might seem a bit surprising at first glance can occur, as I can illustrate by a personal anecdote: During my PhD I gained some experience in functional analysis and operator theory, and briefly after I started my first postdoc position, a PhD student from stochastics knocked on my door: he was trying to solve an inverse problem in statistics which was given by a linear integral equation, and somebody had apparently told him that I'm the linear operator guy. Fortunately, I had time (or, say: I took the time), so we discussed the issue in several meetings and finally resolved it. So in the end we wrote a joint paper, together with his advisor, that mixed up functional analysis and mathematical statistics. What do we learn from this anecdote? Well, if a problem occurs in your research that seems to stem from another field, it can be a good idea to just knock on somebody's door. And if somebody knocks on your door, it can be a good idea to take some time for them.
People from other institutions, part 1: visits at your university.
Researchers usually travel a lot, so it will probably happen quite often that people from other institutions come to your university to give a talk or/and to stay for a (often short) period where they work face-to-face with people from your institute.
These researchers might have been invited by your advisor, or by somebody else, and their visits can be a good opportunity to find collaborators.
How to start a collaboration with a visiting researcher:
Well, I'm beginning to repeat myself: Talk to them, ask them questions, discuss. Most visiting researchers give a seminar talk, so attend such talks; if you find the topic of the talk interesting or useful (let alone both), ask them questions about it (after the talk, at tea, during lunch - often there's also a joint dinner after a talk, so you can go there, too). If you want to discuss even more, just knock on the research visitor's door. Most people are happy when they note that somebody is interested in their work. Besides all this: $(*)$ applies, of course.
People from other institutions, part 2: visit other institutions yourself (if possible).
Some PhD students have the possibility for research stays (a few days or even a few weeks) at other institutions. Since such visits are planned in advance, your host will expect you and will be prepared to spend considerable time to work with you, so this is a great chance for collaboration.
How to visit other institutions.
Of course, it depends on whether your institution (or your advisor) has sufficient funding. If funding is avaible, your advisor is most likely to know good opportunities for a visit or a research stay. Your advisor knows your research topic(s), and she/he knows other researchers in the field, so she/he might ask them whether it is possible for you to visit them.
If your advisor does not suggest a research stay on their own - just ask.
People from other institutions, part 3: conferences and workshops.
Of course, you can meet a lot of interesting and clever people there, and it is a good opportunity to find collaborators. However, many (not all) young academics tend to have a few misconceptions about conferences, so here is some advise:
How to find collaborators at a workshop / conference:
First, and most importantly: A conference is a social event! This means, it is not all about the talks - on the contrary, it is about the talks only as far as talks are social interactions themselves. More concretely, this means (among other things):
If you attend a talk, ask questions! Do not be afraid to ask stupid questions. (In contrast to what some people claim, there do exist stupid questions, but the point is that they do no harm at all.) I tend to ask many questions, and a considerable fraction of them turns out to be stupid afterwards. But I learn from both types of questions (the stupid and the good ones), and people tend to remember the good ones much better than the stupid ones (at least I believe so).
The coffee breaks are important! Do not waste them with the preparation of your talk (instead, have your talk prepared before the conference commences), or with reading a paper, or checking emails. Go drink coffee and talk to people (I personally don't like coffee nor tea, so I eat cookies instead). Ask the speakers additional questions there if you found their talk interesting; if you gave a talk, be there to give people the opportunity to ask you questions.
This way, you will meet many people, and you will notice that you share common interests (and expertise!) with some of them. If this happens: Keep in touch. After the conference, continue discussions that were interesting via email, or via video calls. Again, $(*)$ applies (this time by means of electronic communication rather than face-to-face).
Often collaborations happen by mutual interest ― you start talking with someone, an interesting question comes up, and you decide to pursue it. I don't think I have ever attempted to find a coauthor just for the sake of finding a coauthor (but that doesn't mean you shouldn't do this). Most of the time, it's either with someone I already know really well (this should be easy to initiate!) or with someone who has worked on very similar stuff (which is also not so hard). It's easier if you're generous sharing your own (half or full) ideas.
Just like not all of my own projects end up working out, not every collaboration ends up working out. For example because you get stuck or someone loses interest (is overwhelmed by other work, etc). That's ok; don't beat yourself up or try to force something. But do make an effort.
You might find it easier to be the initiator (coming up with the idea and finding someone suitable), or you might find it easier to join someone else's project. In the latter case you could attend a lot of talks and show interest in other people's work (especially if you are indeed interested!).
If you want to find a coauthor, go to a pub or a conference dinner, find a nice person, use a pick up line, something like"Do you know that the derived category of an abelian category is also abelian?", After you find common interests, you propose " Would you be my coauthor?". If the answer is "yes" you are done. If the answer is"no way" or "I already have a coauthor", find some other person or go to another conference dinner.