Question 1. A paper was rejected because of 'not suitable level of rigour' without a single example of a mathematical error/imprecision. What can the author do in this situation?
It sounds a matter of fact: writing that a work is not mathematically rigorous requires at least one example of a mathematical error/imprecision - for instance, an inaccurate definition. What can the author do, if this rule has been violated?
Needless to say, the author should first recheck the whole work once again. Assume this has been done and no errors/imprecisions have been found. Assume also that the editors refuse to clarify anything.
An instinctive reaction could be then to submit to another journal. But the most probable outcome is an automatic rejection - most likely, the same people are going to handle the work and think: 'oh, that is the same erroneous work, not even corrected'.
The author seems to be in a stalemate.
One can report the case to SciRev: this will not help her or him but might help others.
Question 2. Can you share any of your own experience?
Positive or negative cases are welcome, especially transparent and refutable ones, that is, supplied with a complete correspondence with a journal. Here is just one. (Ethics of publishing received peer reviews.)