This is tricky, because it strikes at the tension between pressures from the publication system to keep papers short vs our collective desire to have a peer-reviewed literature with no gaps. Let's think about what's driving those tensions before we dive into what's the best course of action.
Pressures to keep papers short come from:
- Journals, because there's often a higher bar for longer papers, because they take up more space in the issue.
- Time, because referees take more time when sent a longer paper.
- Jobs, because, all things being equal, a series of short publications is looked upon more favorably than one long preprint.
On the other hand:
- Sometimes the appendix is long simply because of the nature of the proof, and cannot reasonably be shortened or split into separate papers.
- Depending on the referee, it can take the same amount of time to referee a series of papers, because referees for the later ones want to also read the earlier ones to be sure they can be relied upon.
- It's important for results to be peer reviewed and published. If you leave a chunk of the proof on the arXiv but the published part of the paper needs that work, it can raise questions from the reader. People might be unsure if the proof was fully checked, and might wonder why this chunk wasn't published.
When faced with hard questions, it's good to be guided by what's best for the reader while also thinking about what's best for your career. It's clear that, if the paper can be split into a series of short papers, that will make it easier for the reader to digest them in small chunks, and will be best for your career because it's more publications and generally shorter referee waiting times (so, I support "possibility 4" if you can do it). But let's assume the proof cannot be broken up in this way. In that case, I think it's best to publish the one long paper, where half of it is this technical appendix. That is, I support option (1). I also think the advice in the comments to communicate with the editor is good advice. Editors understand the importance of creating a body of literature that avoids relying in critical ways on unpublished proofs. Whenever possible, it's good to have the editor on your side. If you get the editor in your corner, that removes the main obstacle. Your editor can then argue on your behalf to the editorial board if anyone raises an objection about the length. If you are at a stage of your career where you need publications quickly, the editor can also lean on the referee to do their job faster. It would also be smart to aim for journals that publish long papers and aim for electronic journals where length is less of a concern.
One last comment is that sometimes junior mathematicians don't know what bits of a proof can be left to the reader. If something is straightforward, and the author carefully checked it, sometimes it's ok to leave the proof to the reader as an exercise, especially if you think that's important for the reader's success later on in the paper. These kinds of things are not "gaps in the literature." But, I can't imagine leaving 50 pages worth of exercises for the reader. In this answer, I assume the author knows which things are essential to include and which details could be left out, and has decided that all 100 pages are necessary to convincingly prove the result.