A more pernicious instance of appeal-to-authority, of course, would be less open, and not easily checkable. Although such appeals sometimes occur in mathematics, I believe that most mathematicians do not regard them as constituting proof. And so I don't think that we actually have a problem with this. Rather, when a mathematicians says that so-and-so famous mathematician asserts $X$, then I think that this by itself is rarely taken as a proof of $X$. Instead, it is taken as an indication that there likely is a proof of $X$, and that perhaps that famous mathematician has such a proof, and that perhaps we might search for such a proof ourselves.