My personal feeling is that not making the author wait for a second referee to write a report is doing them an enormous favor, even if the paper gets rejected. If you aren't going to write the paper a good enough report to get the paper published, most probably whoever else would referee it instead won't either, and they will take a lot longer.

I have to admit, I also feel like you're not respecting the time of the anonymous person who will have to referee the paper instead of you. Peer review takes up a lot of mathematicians time; it seems like just making the problem worse to create unnecessary duplication.