I still think this question is too personal for MathOverflow and that advice from complete strangers should only be taken with a very serious grain of salt -- evaluated as to whether it really rings true to your actual situation. But given the series of comments so far, including my own, and that the question was reopened, I guess I'll add a few points.
I completely agree with Henry Cohn's (hidden from sight) comments that working on math itself can be very social -- it's good to find collaborators and mentors who are people you like to be around, and it is good to find study groups in school for learning material. People vary a lot in how collaborative they are, probably in part because as you observe people vary in how introverted/extroverted they are. Some people do almost all of their work in collaborations and develop close friendships with their collaborators in the process.
I also very much agree with Emerton's discussion of how cultures in graduate programs can vary a lot and that it's important to find one that is a good fit for you.
Mainly I wanted to say something about your comments that others seem to get things more easily, while you have to study proofs for many hours. I've seen it happen many times that students believe the other students all understand things better and more easily than they do when in fact this just isn't the case at all -- some people have a lot of bravado, throwing fancy words around and pretending they understand things they don't really understand, while others tend to focus all their attention on the things they don't yet understand and ignore all the things they do understand, figuring they don't need to worry about those. Of course I don't know if this is your situation, but it is a possibility to consider as to what might be going on. I think honesty is really valuable in making genuine headway in math -- including working with people where you are comfortable enough that you can be honest with each other about what you do/don't understand. If students around you are acting like they understand proofs right away, one distinct possibility is that they haven't actually thought through all the subtleties and don't actually yet have a full understanding; another possibility is that in fact they are secretly working very hard, only acting like it is coming easily. One excellent mathematician said to me a couple years ago that he thinks hard work plays at least as big a role in success as raw talent. I think there's something to that, along with the importance of finding a good niche for oneself.
I have no idea whether you should continue in math, since I don't know you, but those are a few general thoughts, for whatever they are worth. I hope people will refrain from asking very personal questions on MathOverflow, especially considering what a huge readership it seems to have, but I do wish you good luck.
