Yes and no. I used to do so religiously, but often my notes would digress. I recall a talk on Kirby-Siebenmann in the early 1980s in which I got lost early and drew a vacuum cleaner. More recently, I took notes of selected speakers in a couple of conferences because I wanted to learn the stuff. When the speaker is really uninteresting, it is helpful to take notes if only to stay awake.

On the other hand, there is a mystical phenomenon that occurs when notes are not taken. To give an example, suppose that a speaker is talking about a general Lie group and algebra, and the listener is thinking explicitly about SU(2). The speaker says a generality, and the listener checks the specific case. This continues for a while and the speaker goes on one track while the listener thinks about an entirely different problem. Towards the end of the hour, if both have done their jobs well, speaker and listener come to the same conclusion. Sometimes having a pad of paper handy is good if only to do the calculation that the speaker says is trivial, and the listener can't fathom in the 30 seconds that pass.