You do not want to alienate professors in your department either, so I would say let it be published and forget about it. I don't think you are required to cite it in your CV. After all there is no fixed format for a resume; you put in whatever you estimate highlights your positive features.

As for people judging you from having read the paper, it seems there is little chance of that given that you are not going to work in that 'toxic" field later on. Also, your referee will probably not be a Fields medalist, so there should be no big harm from that quarter either.