Perhaps I should clarify this "answer" in light of the downvote as well as the OPs comment. 

I don't intend to cause offense or make light of your situation. I went through a similar enough phase as a graduate student, and with high probability, so did everyone else who is saying "of course you should go and give talks wherever people let you". So let me point out three concrete and (I hope) helpful things for the next time you are in this situation.

1. One of the signs that you have "made it" in the math business is that you start getting invited to give colloquium talks. These are far scarier to plan than even numerical analysis talks, because there is almost nothing you can say that the whole room will "get"  even though it may be filled to the brim with experts. This is an inevitable outcome of specialization -- one person's basic facts are another person's alien magic. The sooner one gets used to adapting to such situations (both as a speaker and as a listener), the smoother this transition becomes.

2. Typically, you have access to what people are going to say before they say it. At the end of the day, we are all responsible for our own ignorance -- read the introductions to the appropriate papers/preprints of the people who are giving talks in your alien conferences whenever possible. The less blind you walk in, the less blind you'll walk out.

3. And finally, if all has gone to hell and you are completely lost, you can at least focus on the way the speaker is conveying information rather than the content. Watching other people give good/bad talks is a spectacular way of finding your own voice and learning what to do/avoid in your own lectures.

Good luck, and sorry about my tongue-in-cheek knee-jerk response from earlier. I think you will find that these things have a way of working out.

----

I am an undergraduate student. Occasionally for some reason I am asked to take a class at my institution whose stated purpose is almost completely unrelated to my major. To preserve my anonymity, I won't say what I actually do, but as an example: Imagine I am majoring in computational biophysics, and I am asked to take a course in linear algebra.

In the past when I have done this, it has not been a pleasant experience. I sit through a bunch of lectures I don't understand, try to socialize with a group of people who already know each other and who I'll likely never see again, and then take my finals, which I feel is seen as irrelevant.

But I am still encouraged (by my advisor and others) to take such classes, because it is good for my career. In particular, I've been told that it's good to learn diverse techniques and ideas, and that I may someday need them in my career. This seems very unlikely to me, but I am uncertain.

Is taking such classes really a worthwhile endeavor?