Ask yourself: do you want to be part of a community, or do you want to do math by yourself in the woods?

There is no wrong answer: doing either is a fine and potentially rewarding lifestyle, but giving up on the community means giving up on both the positives and negatives that it brings, and most answers here make clear that the positives outweigh the negatives.