Like Konrad Swanepoel, I have found many mistakes, especially in my own work, around "Of course" or comparable expressions, and the saying from one of my early teacher that I often quote is "If it is obvious, then it is easy to prove, so prove it."

That said, I think there are instances where "of course" adds some value to a mathematical text: namely to justify to the reader why you are not taking a seemingly shorter route. Suppose you want to prove a certain assertion, and that in your context, the proof would be easy if some groups were finite, and suppose that in the standard historical paper on the subject that you are generalizing, the groups in question are finite. Then, I think it is helpful to point out at the beginning of your proof or in a remark that "Of course, if the groups were known to be finite, we could use the strategy of...". This "of course", far from drawing attention to how smart the author is, assumes that the reader might have anticipated a seemingly shorter proof and explains why this short route was not taken.