Assume we are talking about a good journal with a large editorial board representing a wide scope of mathematical interests. I will describe both the role of the editors and the role of the referees. This is my personal viewpoint and others might have different opinion/experience.
The role of the editors.
Good journals can accept only about 20% of submitted papers. This is not easy to reject 80% of papers and it often results in rejecting really good papers. Everyone understands that. The procedure of evaluating the papers by the editors is more or less as follows:
The editors have many years of research experience and (hopefully) developed a good mathematical taste. If the paper is close to the research interest of the editor then he or she can relatively easily identify the papers that are not particularly interesting. The reason for not being interesting can be based (for example) on the following criterion:
The result is not well motivated. It is very technical and follows more or less standard arguments. The authors simply take a known result, and prove a new result by slightly modifying the given assumptions. Usually it means that they make the statement more complicated and in a sense more general. Often, they neither have interesting examples supporting such generalizations nor indication of possible applications.
Unfortunately, most of the papers fell into this category. If the editor is sure that this is the case, then he or she rejects the paper without sending it to a referee. Then the authors usually get a rejection notice similar to this one:
We regret that we cannot consider it, in part because at present
we have a large backlog of excellent articles awaiting publication.
We are thus forced to return articles that might otherwise be considered.
If the editor is not sure about the quality of the paper, then they ask an expert (or several experts) for a quick opinion:
I wonder if you could make a quick, informal assessment of it. Are the results strong enough to warrant sending the article to a referee? Because of our backlog, we like to send to referees only articles that appear to be of very good to outstanding quality.
In that case the expert evaluating the paper is not asked to check all the proofs but to make a judgement based on the criterion explained above. This is an easy task for an expert. If the expert writes a negative opinion, then the authors receive a rejection notice often phrased the way as the rejection notice listed above.
For top journals all experts have to write a positive opinion before the paper is sent to referees.
If the experts' opinion is positive, then the paper is sent to a referee or to many referees. The most extreme case that I know of was a panel of 12 referees who took several years to evaluate the paper (this was when Thomas Hales proved the famous Kepler conjecture). For top quality journals all referees must write positive reports before the paper is accepted (once I received 6 reports, 4 positive and 2 not so positive and the paper was rejected).
Let me also add that the editorial boards are structured basically in two different ways. (1) The authors are asked to choose an editor from the list of editors and submit the paper directly to them. Then the editor who receives the paper handles the submission process according to the rules explained above. (2) The authors submit the paper to the main editor or just to the journal and then the main editor either rejects the paper by themselves or he/she sends it to one of the editors from the editorial board and that editor applies the rules listed above.
Of course some of the journals might have a slightly different approach than the one explained here. There is no a canonical solution and what I wrote is a somewhat a simplified version of the process that is applied in reality.
The role of referees. A paper passed through an initial screening and it was sent to a referee. This is the most unpleasant part of the process. A referee spends a lot of time to read the paper, they are not paid for this job and since their work is anonymous, they do not get any recognition for what they do.
What is the referee required to do? First of all, the referee has to assess originality of the results and whether the results are interesting enough. This part is the same as the one in the initial screening when the paper is sent to an expert for a quick opinion. Secondly, the referee is required to read the paper and check details. Let's be clear about that. Unless the paper is directly related to the research of the referee and he or she really wants to understand the details, there is no way the referee can check all details. Since I cannot speak for other people, let me say what I do in this situation.
My answer will only be a simplified version of the real process of the refereeing a paper, just a main idea of what I do.
I go through the whole paper (or most of the paper) to have a good idea of what it is all about, to see a big picture not only of the meaning of the theorems, but also a big picture of the techniques used in the proof. Then I check carefully details of many/some arguments while for other arguments I briefly skim over. If the argument seems reasonable and believable to me I do not bother checking it very carefully. If all details that I check are correct and if all other arguments seem reasonable I am content. In this case, if I like the statement of the main result, I accept the paper. If however, some arguments seem fishy to me, then I check them carefully. This is a point where often I ask the authors for further clarifications. If I really cannot pass through the paper, because I think it has mistakes or if it is written in an unreadable way, I often reject the paper.
The biggest problem is when I am convinced that the result proved in a paper is of an outstanding quality, but the paper is very difficult and for that reason not very easy to read. Then, hmm... Then, it is not easy and I often struggle with making a right decision.
