First of all one should agree on what a substantial error is. Of course, if some or all theorems in a paper turn out to be wrong, this is pretty substantial. But it can also happen that the main results are true as stated, but the proofs are incorrect or incomplete, and then one is confronted with the problem: where to draw the line between substantial and not. I am not proposing a solution to his problem. In fact, I don't think there can be any simple solution: this should be considered on a case by case basis. However, in my opinion the guideline should be: an error or a gap in a proof is substantial when it can't be corrected with the methods used in the paper. This may sound vague but in some cases I'm aware of it was clear what this means.

Now supposing an error has been found, it is for the author to take steps such as publish an erratum or to update the paper on the arxiv or to put a note on the arxiv explaining what is wrong. Now the really difficult case is when the author acknowledges the error but flatly refuses to anything about it. (I'm aware of a couple of such cases; no names will be named.) In this case again there is no universal recipe, I believe, but continue negotiating with the author, editor etc. As a last resort, if everything else should fail, I guess one should put a note on the arxiv explaining that the results and/or proofs of the paper are incorrect, especially when it comes to results other people are likely to use.