Last (academic) year I organised a video seminar. We didn't use Skype; we used some gadgetry called an "Access Grid room". This is some kit which is installed in a lot of universities in the UK and, I believe, across the world. Ultimately at least 20 universities in the UK were capable of connecting to one another and listening to the speaker.

http://www2.imperial.ac.uk/~buzzard/maths/videoseminar.html is a link to the seminar.

Here is a summary of how it worked: the speaker had to be in an Access Grid room. If the audience wanted to be seen and heard, as well as seeing and hearing, they also had to be in an Access Grid room, although this Access Grid room did not of course have to be the same one as the speaker. Access Grid rooms with people in were videoed and the streaming video from each room, plus sound, was broadcast in each of the other rooms (typically on the back wall, so the speaker could see his audience, and then, behind them, 10 more audiences). However, if you just wanted to lurk, you could do so from the comfort of your own home, as long as you had a broadband connection. This is perhaps not quite what the questioner had in mind---for example with skype you'd presumably be able to contribute from home---but it's just a data point to suggest that stuff the questioner is suggesting is already happening.