You could also ask yourself if do not have another type of benefit from lectures. For me it is often not the case that I understand much during a lecture, but just the fact that I see someone who knows the subject handling it live before my eyes changes a lot:

To see what parts of an unknown matter the speaker emphasizes or labels as deep, and what is standard facts, or useful knowledge imported from other areas (and thus not in the focus of the story being told) makes a big difference for me when trying to understand the stuff later. When I sit down to study, I then have the feeling that I am following a road which will lead somewhere, which is often not the case when I just have big text in front of me. Having made the experience that someone could give an outline within, say, one hour also just makes it less intimidating. Also, even if after a talk I could not repeat a single line of what the speaker was saying, I often find myself surprised by being able to come up with details from that talk a year later - I think just the fact that the mathematics comes with a voice, gestures, a handwriting and all other kinds of perceptions makes it stick better in my long term memory and thus gives it a better chance to fall into place later.

If taking notes distracts you from this kind of benefit, you might consider just copying someone else's notes, and paying attention to the whole situation during a talk...